Data, search & knowledge

Web scraping & crawling

676 agents · most starred first

Web scraping & crawling, page 2 of 7

just-scrape★ 76

CLI for AI-powered web scraping, structured data extraction, web search, multi-page crawling, and scheduled page-change monitoring powered by the ScrapeGraph AI API. Turn any URL into markdown, HTML, screenshots, links, images, summaries, or brand assets; pull structured JSON with a prompt + schema; watch pages on a cron with webhook alert

snag★ 75

Snag web pages like a polite robot with a browser

facebook-page-info-scraper★ 71

Free Facebook pages MetaData Scraping Library - Unlimited Calls

satgpt-app★ 69

SatGPT: natural-language satellite imagery search and viewing (LangChain + Panel, 2023). Live recorded demo at kevinlalli.me/satgpt-app

chat-plugin-web-crawler★ 67

🧩 / 🕸 WebsiteCrawler - This plugin automatically crawls the main content of a specified URL webpage and uses it as context input.

perplexity-ai-export★ 64

Grabs all your Perplexity conversations data, spits it out into a nice file folder structure and allows you to find your old conversations again.

scrapit★ 61

A (really) easy way to web scrape

GcrawlAI★ 60

Turn any website into clean, LLM-ready data. Open-source web crawler with stealth mode, distributed crawling, real-time WebSocket progress & Markdown output. Power your AI apps with GcrawlAI.

slither★ 58

A simple, easy to use framework for adding randomized, anonymous IP addresses and user-agents to web scrapers, crawlers, and penetration testing solutions

site-md★ 58

Serve clean Markdown from your Next.js site to AI agents, crawlers, and LLMs. Humans get HTML, agents get clean Markdown of the same pages. Two-file install.

eGet-Crawler-for-ai★ 57

Web scraping framework built for AI applications. Extract clean, structured content from any website with dynamic content handling, markdown conversion, and intelligent crawling capabilities. Perfect for RAG applications and AI training data pipelines. Features async processing, browser management, and Prometheus monitoring.

PythonScrapyBasicSetup★ 56

Basic setup with random user agents and IP addresses for Python Scrapy Framework.

jevfetch★ 56

A web-reading tool for LLM agents that returns the parts of a page relevant to a question, verbatim

supacrawler★ 54

Supacrawler's ultralight engine for scraping and crawling the web. Written in go for maximum performance and concurrency.

firecrawl-quickstarts★ 53

A collection of cookbooks to help developers get started quickly with the Firecrawl API.

crawl4ai-skill★ 51

Scrape JavaScript-heavy sites and extract structured data via reusable CSS schemas. Portable agent skill wrapping the Crawl4AI CLI and Python SDK.

ai-agent-data-apis★ 51

A mega collection of 2,070 AI agent data APIs: MCP servers, LLM tools, RAG pipelines, live web data, vision & voice. Give your AI agent live web data.

video-batch-download★ 50

批量下载自媒体视频并在本地转写文案的 AI Agent Skill,支持抖音、B站、快手、小红书和微博;无需登录,自动获取无登录态的最高画质,可集成任意 AI Agent。

GetbijiEx★ 50

一键导出Get笔记订阅的博主内容,并生成MD文档,并支持Agent skill

UserAgent-list★ 50

Automated User-Agent database for 100+ browsers on 100+ operating systems. Updated every 5 minutes via GitHub Actions. Weighted by real-world popularity, grouped by OS and browser, with a cumulative ALL/ folder of unique agents.

mtywatch★ 48

一句话监控网页内容变化,AI | 爬虫 | 网页监控 | 网页更新提醒 | 网页内容订阅

hermai-cli★ 48

Open-source engine that turns any website into API for AI agents.

wx-kit★ 48

把微信公众号文章下载为多种格式并在应用内浏览;支持直接订阅公众号,每日自动更新公众号文章至本地,同时提供CLI 供 AI agent 调用。

AutonomousGIS_GeodataRetrieverAgent★ 47
sparkfetch★ 47

🔥 Turn any URL into clean, structured, LLM-ready content. The open-source web fetching & extraction API.

crawlerdetect★ 45

🕷CrawlerDetect is a Python library designed to identify bots, crawlers, and spiders by analyzing their user agents.

read-buddy★ 45

Read Buddy: Agent Skills for reading webpages, RSS, YouTube, X, Feishu docs, OCR, podcasts, topics and personal knowledge sources

maw★ 44

Crawl any website into a single searchable file. Query it forever, offline.

peek-api★ 44

Discover internal APIs from any website. Captures XHR/fetch calls, extracts auth headers, outputs structured endpoint catalogs. Like opening Chrome DevTools Network tab, but automated and AI-agent friendly.

openai-crawlers-ip-ranges★ 43

OpenAI crawlers IP ranges. Here are the complete and updated lists of OpenAI IP addresses

Web2BigTable★ 42

A Bi-Level Multi-Agent LLM System for Internet-Scale Information Search and Extraction

Reddit-AI-Agent★ 41

Reddit AI Agent is an intelligent tool that helps you explore Reddit like never before! 🔎 It allows you to search for any query and fetch top Reddit threads along with their most relevant comments. 🚀

GPT-1984★ 39

logs into chat.openai.com bypassing the captcha, write a prompt and print the response

scrapegraphai-ai-copilot★ 39
gaokao-vault★ 39

gaokao-vault, scrapling抓取阳光高考数据,后续会增加其他类似网站的抓取

x-scraper★ 38

A Twitter/X scraper built with Playwright for browser automation and OpenAI GPT-4 for AI-powered tweet analysis. Features timeline scraping, historical search, keyword search, checkpoint/resume, proxy support, and comprehensive data extraction.

Alibaba-CLI-Scraper★ 37

Create your own Alibaba dataset and interact with it in plain English. ⚠️ Archivé — voir poneoneo/pickmysupplier

weibo-chat-auto★ 36

微博群聊消息自动归档 · 原生桌面应用(应用内扫码登录)+ 本地可视化查看器,含 AI 每日摘要与 Agentic 问答

AgentWeb★ 36

Free, dependency-light Internet intelligence MVP for grounded research, extraction, citations, and page monitoring.

ultimate-scraper-agent★ 36

Research skill and researcher subagent for Claude Code: multi-engine search behind one interface, parallel agent waves, and a verify gate that blocks any citation without a verbatim quote. No personal data; configured via .env.

scraping-agent-ai★ 35

AI-powered web scraping agent built with LangGraph, LangSmith, Firecrawl, and Anthropic AI. Automates intelligent crawling, structured data extraction, and LLM-powered content formatting. Efficiently handles anti-bot mechanisms, error recovery, and batch processing. 🚀

scraping-agent-skeleton★ 34

Research and scraping agent skeleton: tool loop, loadable skills, fallback chains. No data included, configured via .env.

Smarter-Web-Scraping-with-Python★ 32

Leverage modern open-source tools to create better web scraping workflows.

gptauto★ 31

ChatGPT selenium scraper written in Python

signup-forms★ 31

Code for the paper "World of Bits: An Open-Domain Platform for Web-Based Agents"

SouWen★ 31

Unified search, crawling, and archiving toolbox for AI agents and automation scripts.

Wikipedia-Scraping-with-LLM-Agents★ 30

Scraping Wikipedia by combining LangChain's agents and tools with OpenAI's LLMs and function calling

git-repo-parser★ 30

A tool to scrape all files from a GitHub repository and turn it into a JSON or TXT file, Useful for AI and LLM Projects

zhihu-toolkit★ 30

Zhihu scraper and Markdown conversion utilities.

AI-web_scraper★ 29

Just mention want you want and it will extract/scrape data from the Web. Useful to create AI web search+extraction/scraping agent, RAG with web data etc.

docpull★ 29

Local-first Python CLI and MCP server for versioned, cited context from web and document sources.

tourist★ 29

Open-source, LLM-ready SERP and web scraping service

llm-low-cost-search★ 29

Async Brave Search + Readability extractor for LLM/RAG — 10–20 links per request, batch extraction, JSON archiving, typically 1/8–1/5 the cost of hosted stacks.

Reddit-Content-Research-Agent★ 28

Build a Reddit Content Research Agent with LLMs, LangChain, SERP, Jupyter, Django, Bright Data, Celery, Django QStash, and much more.

web-crawling-guides★ 27

How to guides on web-crawling or scraping

GPT-3.5-ON-STEROIDS★ 27

GPT-3.5-ON-STEROIDS combines GPT with Python tools, empowering dynamic web scraping, language processing, and data retrieval. Contribute to advancing text generation with AI. 🚀

spider-clients★ 26

Python, Javascript, and Rust libraries for the Spider Cloud API.

openai-scraper★ 26

This is a template repository for building a web scraper with OpenAI support. The repository provides a basic project structure with TypeScript and Puppeteer pre-configured, as well as OpenAI's GPT-3 API integration. With this template, you can easily build a scraper that uses machine learning to analyze and extract insights from the scraped data.

TackleAI★ 25

Extract information from any websites using openAI

pricing-page-scraper★ 25

Parse SaaS pricing page using Open AI - GPT-3.5

nginx-markdown-for-agents★ 25

Self-hostable NGINX module that serves Markdown to AI agents via HTTP content negotiation (Accept: text/markdown), powered by a Rust converter.

raggy★ 24

scraping and querying documents for LLMs

deepspider★ 24

AI 原生智能爬虫与 JavaScript 逆向工程平台,基于 DSH、Patchright/CDP 与独立语义运行时,从浏览器证据恢复参数生成逻辑并交付可验证 Solver。 | AI-native web scraping and JavaScript reverse-engineering platform powered by DSH, Patchright/CDP, and an independent semantic runtime—from browser evidence to recovered parameter-generation logic and verifiable Solvers.

markdown-for-agents★ 23

HTML to Markdown converter for AI agents. 90+% fewer tokens, one dependency, works everywhere.

openclaw-ultra-scraping★ 23

🕷️ Adaptive web scraping skill for OpenClaw agents — bypasses anti-bot, survives site redesigns. Powered by MyClaw.ai

mark-it-down★ 23

Convert any web page to clean Markdown

PerimeterX_RE★ 22

Complete reverse engineering of PerimeterX (HUMAN Security) anti-bot SDK · pure-algo _px3/_px2 generator (no browser, no Selenium) · iFood + Grubhub + Bundle 10/10 verified · WASM PoW solved · 68 production gotchas · AI Skill included.

web-extract-with-chatgpt★ 21

A Python project that extracts data from websites with the option to process the data through @openai's ChatGPT API. The results are either printed to stdout or sent via a POST request.

openai-structured-outputs-with-firecrawl★ 21

This repository demonstrates how to leverage OpenAI's GPT-4 models with JSON Strict Mode to extract structured data from web pages. It combines web scraping capabilities from Firecrawl with OpenAI's advanced language models to create a powerful data extraction pipeline.

wechat-reader★ 19

面向 AI Agent 的微信公众号文章阅读工具。提供 CLI、MCP server 和 Python API。

GPT-auto-webscraping★ 18
AgenticCrawler★ 18

acrawl — LLM-powered web crawler. Describe what you want in plain English, get structured data back. Single Rust binary, 25 providers, MCP server built-in.

fetchrouter★ 18

FetchRouter - Multi-agent web scraping system for Claude Code

langchain-scrapegraph★ 17

ScrapeGraph client langchain integration

llmweb-rs★ 17

Webpage to structured data in Rust & LLM

mojo★ 16

Non sucking cross-platform extremely fast C++ crawler to convert entire websites into LLM readable data

GPT-auto-webscraping★ 16

streamlit region:us

scraper★ 14

RAG-based Web Scraping

DocsScraper.jl★ 13

Efficient RAG knowledge pack creator from online Julia documentation

webtap★ 12

AI web scraping python library for efficient and reliable web scraping.

url-to-markdown★ 11

Convert webpages to clean Markdown for LLM and RAG workflows. Browser-based UI + Node.js CLI with selector drilling, metadata extraction, and batch processing.

web-crawler★ 11

Scrapling 风格隐身爬虫库:自适应选择器 · TLS 指纹 HTTP · JS 渲染 · 回调式 Spider 框架(重试/robots/去重/中间件管道)· JS 逆向 Agent · 验证码识别 · MCP 服务 · 本地 Web UI · 授权渗透侦察 · 默认 SSRF 防护

ai-docs-vector-db-hybrid-scraper★ 10

Retrieval-augmented docs ingestion stack: Firecrawl + Crawl4AI + Qdrant vector search with FastAPI and MCP interfaces for AI engineers.

next-markdown-mirror★ 10

Self-hosted Markdown for AI agents — serve Markdown instead of HTML when AI agents request it

Reddit-to-AI★ 10

Scrapes Reddit threads and sends content to chosen AI chatbot for analysis.

flyscrape★ 10

The Most Powerful Open-source LLM Friendly Typescript Web Crawler & Scraper

proxyclaw★ 10

Residential proxy SDK for Python. 175M+ IPs across 195+ countries. pip install iploop

pangolinfo-amazon-scraper-cli★ 10

Pangolinfo 亚马逊爬虫与数据采集工具:基于 Pangolinfo Amazon Scrape API / 数据 API 实现 Amazon 实时数据采集(商品详情、关键词、评论、榜单、类目/利基),输出 AI 友好的 JSON/Markdown。面向 Agent 与 AI 应用,支持 Skill 集成,低成本、高效率、易维护的采集软件与采集工具方案。

smart-agent★ 10

5平台数据采集框架 — B站/知乎/快手/微博/贴吧 纯HTTP直连,零浏览器依赖,毫秒级响应

cli★ 10

The official Decodo web scraper CLI for terminal web scraping and AI agent workflows.

omnifeed★ 10

LLM-friendly web crawler & scraper with a dedicated Reddit engine, built on Crawl4AI — Open WebUI compatible

agentcrawl★ 10

Self-hosted web scraping and Markdown extraction for AI agents

AstroProject★ 10

Async web crawler powered by Playwright headless Chromium — JavaScript rendering, AI-powered extraction (OpenAI / Anthropic / Google), CSS rule engine with visual preview, and a full PySide6 desktop GUI. SQLite WAL crash recovery, proxy circuit breaker, RFC 9309 robots.txt, ReDoS-immune regex, 3,822 tests. pip install astrocrawl[full,gui,fast].

TelegramScraper★ 10

Scrape public Telegram channels without API or login — Python CLI, MCP server for AI agents, dashboard

pobear★ 10

Open-source web scraping CLI. Crawl sites, extract data with CSS selectors or Markdown, and fall back to a proxy or the Decodo Web Data API when blocked. Single Go binary, no runtime or account required.

context-dev-cli★ 10

Context.dev CLI: scrape, crawl, search and screenshot the web from your terminal. Returns LLM-ready markdown or JSON.

SocialOSINTLM★ 9

Social Media OSINT providing conversational and holistic view of user activity

crawlkit★ 9

🕷️ Open-source web crawling toolkit — Video, OCR, NLP, Stealth, 10+ parsers

vidilearn★ 9

Production-grade content extraction agent for YouTube and the web. Extract transcripts, clean articles, and structured metadata locally with zero API keys. Features automatic Playwright fallback for dynamic sites, native MCP server mode, and local embedding generation.

cockroach-crawler★ 9

Give AI agents bounded eyes on the public web: crawl, search, and normalize evidence without an unrestricted browser.

More in Data, search & knowledge