Data, search & knowledge

Web scraping & crawling

Web scraping and crawling projects fetch and extract content from websites so other agents can use it as input. Some crawl and convert pages to markdown or structured data, as firecrawl and trafilatura do, while others target specific sources or platforms.

676 agents · most starred first

Top 100 of 676

firecrawl★ 204,752

Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.

Agent-Reach★ 113,459

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

crawl4ai★ 87,593

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

Scrapegraph-ai★ 32,868

Python scraper based on AI

Douyin_TikTok_Download_API★ 21,499

🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

trafilatura★ 7,155

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

llm-scraper★ 6,957

Turn any webpage into structured data using LLMs

AnyCrawl★ 3,577

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

CyberScraper-2077★ 3,303

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

ai-crawler-py★ 3,251

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

spider★ 2,855

Foundational low latency web data collecting in Rust

reverse-api-engineer★ 2,764

Reverse engineer any website into a typed API client. Works with Claude Code, Codex, Cursor and OpenCode.

webclaw★ 2,439

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

Crawler-Detect★ 2,374

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

weibo_terminater★ 2,313

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

boss-zhipin-scraper★ 1,774

Boss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.

ShardBrowser★ 1,650

Free, open-source anti-detect browser launcher for web scraping and multi-accounting. By the ShardX team (formerly ProxyShard). Engine-level fingerprint spoofing in Chromium 157 (WebGL / WebGPU / Client Hints / fonts / TLS), 220+ device profiles bundled, stable QUIC + WebRTC over SOCKS5.

crw★ 1,280

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

reverse-api-engineer★ 1,142

The agent that turns websites into APIs!

donsetch★ 1,124

Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.

ai-scraper-py★ 1,088

AI Scraper is a powerful scraping tool and scrape agent built to automate data extraction with unmatched precision. Ideal for scalable AI scraping tasks across diverse web sources, this tool simplifies complex scraping operations into efficient, intelligent workflows.

x-tweet-fetcher★ 1,012

Fetch X/Twitter tweets, replies, timelines, search, Lists and X Articles without login or API keys. Python CLI + library for AI agents: FxTwitter / Nitter / browser backends with auto fallback, unified JSON, optional SQLite archive. 免登录免 API 抓取推特推文。

crw★ 968

Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

firecrawl-app-examples★ 846

🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

ketch★ 755

Fast, stateless CLI for web search and scrape. Built for AI agents.

oc★ 691

Turn any website into a compact CLI tailored for AI agents. Browse the web in hundreds of tokens, not tens of thousands.

catbus★ 638

一个命令操作小红书、抖音、TikTok、B 站、快手、微博、闲鱼、淘宝、京东、X:统一的命令与 JSON 输出,给人和 AI Agent 用

JARVIS★ 626

JARVIS: a real-time agentic intelligence-gathering platform powered by autonomous web scraping & OSINT, streamed via Meta Ray-Ban smart glasses

harvester★ 612

Intelligent data acquisition framework for GitHub and web sources

reader★ 576

Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.

teemii★ 557

A versatile, self-hosted manga reader and manager with extensible agent-based metadata retrieval

device-detector-js★ 500

A precise user agent parser and device detector written in TypeScript

scraperai★ 497

ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.

AutoScraper★ 493

Official implement of paper "AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation" [EMNLP 24']

markdown-crawler★ 480

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

List-of-user-agents★ 461

List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)

xcrawl-skills★ 447

Reusable Xcrawl skill definitions for multi-agent runtimes, focused on API-first web data workflows.

AI-Search-py★ 421

Turn plain English queries into structured web data, featuring automated markdown extraction and full JavaScript rendering for dynamic websites.

DeviceDetector.NET★ 411

The Universal Device Detection library will parse any User Agent and detect the browser, operating system, device used (desktop, tablet, mobile, tv, cars, console, etc.), brand and model.

top-user-agents★ 384

Always up-to-date list of the top 100 most common browser user-agents for HTTP clients.

google-news-scraper★ 384

Lightweight scraper for Google News

OpenAver★ 367

Free open-source desktop app that scrapes JAV metadata and generates NFO + cover art for Jellyfin, Emby & Kodi. No Docker, no CLI — one-click install on Windows & macOS. 8 built-in sources + optional Metatube federation (30+ providers), actress collections, cross-language tag aliases, and a REST API for AI agents.

uasurfer★ 358

Go package for fast and reliable abstraction of browser user agent strings.

user_agent★ 346

Generator of User-Agent header

extractor★ 322

Use LLMs to robustly extract web data

woothee★ 320

User-Agent parser/classifier for multi languages

agent-fetch★ 316

Full-content web fetcher for AI agents — Chrome TLS fingerprinting, browser impersonation, multi-strategy article extraction, and web scraping.

llm-reader★ 315

Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extraction easy.

reader★ 311

📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an LLM-friendly input with a simple prefix http://127.0.0.1:3000/https://website-to-scrape.com/

jina-cli★ 311

A lightweight CLI tool wrapping Jina AI Reader API for AI Agents. Fetch and parse any URL into LLM-friendly formats (Markdown/Text/HTML). Perfect for reading X(Twitter), blogs, and news sites.

MinerU-HTML★ 302

MinerU-HTML: An SLM-powered HTML main content extractor that outputs clean HTML bodies. Perfect for Deep Research Agents, RAG applications, and training data generation.

digcrawl★ 302

Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.

moore-wechat-article-downloader★ 297

本地优先的微信内容情报库:同步公众号文章,保存精选评论和互动数据,供 Codex/Claude Code 做内容研究。

teracrawl★ 291

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative

pagesift★ 291

Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.

PyScrappy★ 265

Adaptive Python web scraping toolkit + MCP server for AI agents. Self-healing selectors that survive site changes, TLS-fingerprint stealth to bypass anti-bot filters, CSS/XPath parsing, and 24 built-in scrapers, clean, structured, LLM-ready data from any URL.

ah-douyin-clean-downloader★ 257

把抖音链接或口令直接交给 Codex,下载无平台角标原视频并按博主自动归档

search-result-scraper-markdown★ 243

This project provides a powerful web scraping tool that fetches search results and converts them into Markdown format using FastAPI, SearXNG, and Browserless. It includes the capability to use proxies for web scraping and handles HTML content conversion to Markdown efficiently.

lego-ai-parser★ 237

Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

x-twitter-scraper★ 205

X (Twitter) scraper API & X API alternative with REST, MCP, SDKs & webhooks. Not affiliated with X Corp.

x-md★ 203

Turn public X posts, profiles, and search into compact Markdown or JSON for agents. Hosted at x.pcstyle.dev.

rdrr★ 195

Convert any URL to clean markdown for AI agents.

markdown_web_browser★ 190

Renders any URL via headless Chrome, tiles screenshots into OCR slices, and streams structured Markdown + provenance back to AI agents and pipelines

quantumproxies.io★ 186

Residential, datacenter, ISP & mobile proxies with 90M+ IPs in 200+ countries — plus Scraper & SERP APIs. Code examples in Python, Node.js, PHP, Go, Java, C#, Ruby and curl.

AutomatiQ★ 180

Simplify Web Reverse-engineering for AI Agents

chatgpt-well-known-plugin-finder★ 172

Checks Alexa's top 1M websites for the presence of OpenAI's new .well-known/ai-plugin.json files

real-estate-ai-agent★ 161

Intelligent Python system that extracts real estate property data as structured JSON using AI agents, Nebius Qwen LLM, and Bright Data MCP.

media-agent★ 156

Scrape data from social media and chat with it using Langchain

decodo-openclaw-skill★ 152

OpenClaw skill for scraping any URL using the Decodo Web Scraping API.

blackmaria★ 151

Python package for webscraping in Natural language

fitter★ 133

New way for collect information from the API's/Websites

Scrapegraph-demo★ 120

Streamlit demo of Scrapegraph-ai for GPT4-hackaton

web-scraper★ 118

gradio mcp-server region:us

anansi★ 117

A self-healing scraper for hostile sites: broken selectors repair themselves, browser rendering kicks in when needed, and a coherent identity layer (Chrome TLS fingerprints, matched personas, vendor-aware Cloudflare/Akamai/DataDome handling) works to slip past bot detection. Ships with an MCP server so any LLM can drive a full crawl by conversation.

wxpath★ 110

wxpath - declarative web crawling with XPath; a Web Query Language (WQL)

LLMWebCrawler★ 107

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

DataParasite★ 102

A simple yet versatile context engineered for scalable online data collection

amfe-env★ 99

Parsing User-Agent on a mobile device, could detecting what os/browser/app.

prosecutor-database★ 97

Civic Tech & Data AI For Good project. Tracks prosecutor election messaging, mass incarceration indicators, and historical context. Utilizes Agentic AI for data scraping.

free-proxy-list★ 96

Free proxy list, verified around the clock: HTTP, HTTPS, SOCKS4 and SOCKS5 proxies with country, anonymity, latency and uptime on every row, updated several times a day. Live list, proxy checker, keyless API and MCP server at hproxy.com.

scrapegraph-py★ 92

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic browser automation, scheduled jobs, and structured data extraction

WebScraper★ 91

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

ytfetcher★ 91

⚡ Build structured YouTube datasets at scale — effortlessly fetch transcripts and rich metadata for NLP, ML, and AI workflows.

google-browser-scraper★ 91

Self-hosted Google search scraper for Python. Scrape Google SERP results with a real Chrome browser and your own residential proxy. Self-hosted SERP API alternative with JSON output, an HTTP API and an MCP server for AI agents.

robots.txt★ 90

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

groktocrawl★ 89

Self-hosted, API-compatible Firecrawl alternative with Agent endpoint. MIT license. One docker compose up.

UserAgentParser★ 88

User Agent Parser written in Java

IPproxy★ 87

The proxy IP pool, crawling the mainstream free agent, automatically re -processing, automatically testing the availability of the agent, and the common request header has been attached

twitter_user_tweet_crawler★ 87

A Python crawler tool that can automatically simulate browser operations to crawl all users' tweet content and save all static resources (videos, pictures) locally without calling the Twitter API. Use the selenium library.

skills★ 87

51 Agent Skills for the Scavio web data API: one umbrella skill plus one per platform for Google, Amazon, TikTok, LinkedIn, YouTube, Instagram, Reddit and 40 more, as structured JSON

advanced-sitemap-parser★ 86

XML sitemap parser designed to extract and process millions of URLs while bypassing most modern anti-bot protections. Supports plain and compressed XML, unlimited nested sitemaps, multi-threading, multiple inputs, CloudScraper integration, fingerprint randomization, proxy/user agent rotation, auto stealth mode, and detailed monitoring.

ai-web-scraper★ 85

AI web scraper built with Crawl4AI for extracting structured leads data from websites.

laravel-markdown-response★ 85

Serve markdown versions of your HTML pages to AI agents and bots

userAgentLists★ 82

Get your lists of User-Agent Strings here

webforai★ 82

Any HTML to the cleanest Markdown, for humans and AI agents: a TypeScript library, a CLI with an Agent Skill, and a hosted crawl/scrape API.

php-useragent★ 81

A User-agent analyze project which written by PHP.

pulpie★ 80

Pareto-optimal models for cleaning the web — fast, encoder-based main-content extraction from HTML.

geod-global★ 80

遥感影像与地理空间数据聚合下载工具 | Satellite imagery & geospatial data downloads. Sentinel, Landsat, DEM, land cover, climate, vectors, map services + BYOK Agent.

langchain_yt_tools★ 76

Langchain tools to search/extract/transcribe text transcripts of Youtube videos. Some of this has been integrated into LangChain main branch

Website-Crawler★ 76

Extract data from websites in LLM ready JSON or CSV format. Crawl or Scrape entire website with Website Crawler

More in Data, search & knowledge