scraping
Web scraping is the process of programmatically retrieving web pages and extracting structured data from them for analysis, storage, or reuse. It powers price monitoring, search indexing, market research, and training-data collection.
Scraping ranges from plain HTTP requests and HTML parsing to full browser automation for JavaScript-rendered and anti-bot-protected pages. The ecosystem includes request libraries, HTML parsers, headless browsers, and cloud extraction platforms.
Here are 8,906 public repositories matching this topic...
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
-
Updated
Oct 14, 2026 - TypeScript
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
-
Updated
Oct 14, 2026 - Python
Scrapy, a fast high-level web crawling & scraping framework for Python.
-
Updated
Oct 14, 2026 - Python
🕵️♂️ Collect a dossier on a person by username from 6K websites
-
Updated
Oct 14, 2026 - Python
Python scraper based on AI
-
Updated
Oct 6, 2026 - Python
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
-
Updated
Oct 14, 2026 - TypeScript
Pythonic HTML Parsing for Humans™
-
Updated
Apr 16, 2024 - Python
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
-
Updated
Jul 5, 2025 - Python
🦊 Anti-detect browser
-
Updated
Oct 14, 2026 - Python
Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
-
Updated
Oct 7, 2026 - JavaScript
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
-
Updated
Oct 14, 2026 - Python
List of libraries, tools and APIs for web scraping and data processing.
-
Updated
Sep 19, 2026 - JavaScript
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
-
Updated
Jul 29, 2026 - Python
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
-
Updated
Oct 9, 2026 - HTML
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
-
Updated
Oct 6, 2026 - Python
🚀 Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)
-
Updated
Oct 14, 2026
- Followers
- 368 followers
- Website
- github.com/topics/web-scraping