scrapy/scrapy ↗
Created Dec 28, 2020 · View the scrapy/scrapy repository page
Scrapy, a fast high-level web crawling & scraping framework for Python.
Want this for your repo?
Render a free sample of any GitHub repo in seconds.
Contributors
495
Lines of Code
19,787
From
May 4, 2009
To
Dec 21, 2020
About scrapy/scrapy
Scrapy is a comprehensive web scraping framework for Python designed to extract structured data from websites at scale. It provides a high-level, cross-platform toolkit that handles the complexity of crawling websites, including managing requests, following links, parsing responses, and storing data. The framework is built for performance and is suitable for both simple scraping tasks and large-scale web crawling projects.
The project is written in Python and requires Python 3.10 or later. It is actively maintained by Zyte and a community of contributors, with continuous integration testing across Ubuntu, macOS, and Windows platforms. Scrapy is widely used by data scientists, developers, and organizations that need to systematically gather web data for analysis, machine learning, or business intelligence purposes.
The framework features a robust architecture with support for multiple output formats, automatic throttling, cookie and session handling, and extensibility through middleware and plugins. It handles common web scraping challenges like respecting robots.txt, managing concurrent requests efficiently, and dealing with JavaScript-heavy sites through integration with other tools.