Data, search & knowledge

Web scraping & crawling

676 agents · most starred first

Web scraping & crawling, page 3 of 7

link-video-downloader-zhenxiangai★ 9

把视频号、B站、小红书或抖音链接发给 Hermes,在 Mac 本地完成下载、整理和逐字稿;支持视频号、B站、抖音博主批量抓取。

tiktok-shop-scraper★ 9

TikTok Shop Affiliate Scraper. Find creators promoting any product with verified data (product name, price, sales, creator followers). Apify actor.

Zillusion★ 9

Ask in plain language, get ranked real data sources — then an agent builds, validates and runs the scraper for you. The validator cannot rubber-stamp its own work.

url-intelligence-agent★ 9

Evidence-first AI URL intelligence agent for crawling, entity resolution, social discovery, SEO, technology detection, RAG, MCP, monitoring, reports and web intelligence. Built inside the HORNO ecosystem.

DataHarbor★ 9

Open-core data platform with self-healing web scrapers — AI agents diagnose and patch broken extractors automatically (ClickHouse, Dagster, Qdrant)

stackoverflow-scraper-messenger-bot★ 8

A messenger bot that answers messages by scraping stackoverflow questions and answers

llm-scraper-py★ 8

Python implementation of https://github.com/mishushakov/llm-scraper

monster-hunter-now-events★ 8

A tool that auto-generates calendar events for Monster Hunter Now by scraping web news articles, processing them with AI, and creating a convenient calendar subscription.

Crawl4ai-RAG-with-Local-LLM★ 8

A tool for scraping web documentation using Crawl4AI, converting it to Markdown, and preparing it for integration with local LLMs (like Ollama) to enhance their knowledge for learning and "vibe coding" workflows.

scrappe-tout★ 8

Scrappe-Tout is a web scraping tool designed to convert HTML documentation into clean, organized Markdown files. It automatically handles navigation menus, removes duplicate content, and generates descriptive filenames for easy reference.

real-time-market-analysis★ 8

Automated pipeline for real-time market analysis: Scrape live job data, generate structured CSVs, and create analytics scripts using Kiro with Bright Data MCP.

instagram-reels-transcript-api★ 8

Instagram Video & Reels Transcript API examples powered by Apify. Extract timestamped transcripts from public Reels and video posts, process multiple URLs in bulk, and collect Instagram profile post metadata. Integrate with Python, Node.js, Go, Java, Rust, cURL, and AI agents via Apify MCP.

agent-search-cli★ 8

Enable AI agents to search, crawl, and extract web data with IP rotation, CAPTCHA handling, and rate limit management via CLI and Python.

Collecty★ 8

An Intelligent, Agentic Web Scraping Framework for Automated Schema Discovery and Structural Analysis.

Real-Time-Social-Media-Content-Retrievel-System★ 7

The Real Time Social Media Content Retrieval System fetches real-time LinkedIn posts based on user queries, offering multiple post retrieval and customization options. Although initially focused on LinkedIn, it can be expanded to incorporate other social media platforms, facilitating cross-channel post similarity searches.

Automatic-web-scraper-with-LLM-parsing★ 7

This project is an automatic web scraper that uses the LLM Ollama gpt-oss:20b to parse the body content of a web page. The application is built using Streamlit for the user interface and various Python libraries for web scraping and parsing.

Scrape-AI★ 7

Scrape-AI is a Python library for intelligent web scraping using LLMs like, OpenAI, Anthropicand Selenium. It dynamically fetches data based on user queries, enabling agent-like scraping with configurable models, modular design, and support for real-time data extraction from dynamic websites.

zhihu-lens★ 6

Chrome extension + multi-agent LLM pipeline that bypasses Zhihu's anti-scraping defenses to profile any public user across PII leakage, content authenticity, and HEXACO personality. 绕过知乎反爬,浏览器扩展 + LLM 多智能体生成用户深度画像。

ChatScraping★ 6

Web application to help develop scraping techniques to create context in LLM prompts.

ai-auto-crawler★ 6

ai-auto-crawler: puppeteer + autogen

Deep-Research-With-Web-Scraping-by-LLM-And-AI-Agent★ 6

Use LLM/AI agent for Web scraping (collection data) and analysis data with deep research

crawl-rag★ 6

Web crawler for generating RAG knowledge base files

tiktok-transcript-api★ 6

Instagram Reels Transcript API examples using Apify. Integrations for Python, Node.js, Java, Go, Rust, cURL, batch processing, and MCP workflows for ChatGPT, Claude, and Gemini.

github-scraping-agent-ai★ 6

Project that scape all github repositories, read them, analyse with an AI and return you on a discord webhook project that could interest you.

tubelm-link-picker★ 6

Copy multiple YouTube links into NotebookLM and AI research notes.

RAG-Scraper★ 6

docker anycoder region:us

sdk★ 5

Lightfeed SDK to search and filter web data

Thordata★ 5

> Official Thordata developer portal repository. Curated overview of Thordata's proxy infrastructure, web data tools, SDKs, tutorials, and AI/LLM integrations.

agent-view★ 5

Chrome extension to see the website as AI agents fetch it with Markdown.

WebScraperToolkit★ 5

AI-first web scraping engine with stealth bypass, MCP server, and multimodal output (Markdown, JSON, PDF) for agents and automation.

AI-WebScraper★ 5

An intelligent, universal web scraper powered by Google Gemini AI. Features intent-based data extraction (E-commerce, Leads, News), anti-bot bypass (DrissionPage), and a secure, human-in-the-loop Streamlit interface.

langchain-advertools★ 5

LangChain integration for advertools

crawl-for-vector-db★ 5

A web site crawler for semantic search.

scraper-flow★ 5
torch★ 5

The self-healing AI scraping agent

Web-Context-Retrieval-Engine-for-LLMs-and-AI-Agents★ 5

A modular Python-based search engine pipeline that fetches live web data, extracts both visible and hidden context using advanced NLP techniques, and semantically indexes content for enhanced retrieval. Perfect for powering LLMs and AI agents with up-to-date, relevant context.

twitter-cli★ 5

Access Twitter timelines, bookmarks, and profiles from the terminal without requiring API keys, offering a simple CLI user experience.

scrapfly-cli★ 5

Agentic CLI for the Scrapfly platform: scrape, extract, crawl, and drive a cloud browser over CDP. Built-in LLM agent with Anthropic, OpenAI, Gemini, and Ollama. JSON-first, pipe-friendly, one static binary.

x2md★ 5

Convert an X (Twitter) URL into clean markdown — tweets, photos, videos, long-form X Articles, and top replies. No auth, no API key.

website2markdown★ 5

Convert any URL to clean Markdown. Cloudflare Worker with 14 site adapters, MCP Server, Agent Skills, llms.txt. Open source, Apache-2.0.

AstroCrawl★ 5

Async web crawler powered by Playwright headless Chromium — JavaScript rendering, AI-powered extraction (OpenAI / Anthropic / Google), CSS rule engine with visual preview, and a full PySide6 desktop GUI. SQLite WAL crash recovery, proxy circuit breaker, RFC 9309 robots.txt, ReDoS-immune regex, 3,822 tests. pip install astrocrawl[full,gui,fast].

xpoz-python-sdk★ 5

Python SDK for Xpoz — Pythonic access to Twitter/X, Instagram, Reddit & TikTok data for social listening, research, and AI agents. PyPI: xpoz

citeseal★ 5

Local-first, auditable, agent-ready research archive. Capture fragile web content into structured, verifiable local artifacts that humans and AI agents can consume.

web_scraping_with_scrapegraphai_and_langgraph★ 5

This repository contains an intelligent web scraping solution that uses ScrapeGraphAI for LLM-powered content extraction and LangGraph for orchestrating the scraping workflow. The system can intelligently crawl websites, extract content using natural language instructions, and search for specific information.

mirror-web-cli★ 5

Mirror Web CLI — high‑fidelity website mirroring to clean, offline‑ready HTML/CSS/JS. Handles Next.js images, lazy assets, CSS url(...) rewrites, and robust media capture.

markmaton★ 5

Lightweight HTML-to-Markdown parser for AI agent workflows — Python CLI and API around a fast Go engine, available on PyPI.

skills★ 5

Agent skill for the Supadata API — transcripts, metadata, web scraping, and AI extraction

webcrawler★ 5

Web Data Crawling and Vectorization

markcrawl★ 5

Fast Python web crawler for RAG and AI ingestion. Extracts clean Markdown from any site for LLMs and vector stores.

Crawlit★ 5

Give your AI agent the web: self-hosted Google search, scraping & crawling → clean Markdown. Firecrawl-compatible, free, one docker compose up.

social-media-scraper-skill★ 5

🔍 Agent skill that scrapes any social media post — metadata, comments, transcription & AI video analysis. Drop a link, get everything. Instagram · TikTok · Twitter/X · YouTube

douyin-benchmark★ 5

拆抖音对标 · 输入一个抖音账号,全量采集公开作品数据并产出对标调研报告(原始数据 + Markdown + 浅色杂志风 HTML)。Agent-friendly skill.

md-succ-ai★ 5

URL to Markdown API — md.succ.ai

instagram-private-api★ 5

Unofficial Instagram & Threads mobile API SDK for Python — 60+ endpoints, Android + iOS mobile protocol, zero emulators

agenticschema★ 5

Turn any webpage into typed JSON, callable MCP / WebMCP tools, or AI-native Epistemic Context — in 3 ms, using 2.7× to 28× fewer tokens than a text scraper and up to 1,449× fewer than raw HTML

iranseda-crawler-golang-★ 5

iran seda crawler golang

sample-crawl4ai-bedrock-knowledge-base★ 5

Prepare web content with Crawl4AI and ingest it into Amazon Bedrock Knowledge Bases, with an optional scheduled AWS Fargate (CDK) deployment.

free-proxy-list★ 5

Verified HTTP, SOCKS4 and SOCKS5 proxy data for developers and AI agents. JSON, CSV, TXT and API access with check timestamps, country, anonymity, latency and uptime.

lookacrawler★ 5

Free, open-source, token-efficient local alternative to Firecrawl with native MCP Server for LLMs (>73% token reduction).

antiRea★ 5

Server-side TypeScript library that meters dataset access per identity and flags coordinated bot abuse - bulk extraction crews, scheduler-driven herds, seat squatting. Advisory only.

g2-reviews-scraper★ 5

G2 reviews scraper on Apify — export G2 software reviews with 6 sub-ratings, NPS, switched-from competitor battlecards and LLM-ready markdown. No login; CSV/JSON/API.

Scrapo★ 4

AI-native, agent-first web scraping for Python. Cost-aware tiered fetching (HTTP, browser, stealth, agent), model-agnostic LLM extraction with self-healing selector caches, API-first resolution, deterministic replay and field-level diff, and an MCP server.

Stone_Scraper★ 4

Stone Scraper is an AI-powered tool for automated web data extraction. Built with Streamlit, Langchain, and Selenium to transform unstructured web content into clean, actionable insights. CURRENTLY WORKING ON OPEN AI API CLOUD DEPLOYMENT. Try it out ⬇️⬇️

Scrapper-Enricher★ 4

An AI-powered, multi-agent pipeline that automatically discovers, scrapes, geocodes, validates, and persists comprehensive school data into a Supabase database — orchestrated entirely by CrewAI agents connected to your database via a live MCP server.

GamedataAutoFlux★ 4

一站式游戏数据采集与自动化分析工作流,内置可视化 WebUI 与 AI Agent 助手。

rippr★ 4

YouTube transcript ripper for RAG, LLM workflows, and AI research. Chrome extension. Multiple output formats

hotels-com-reviews-scraper★ 4

Hotels.com reviews scraper on Apify — Hotels.com guest ratings, category sub-ratings, review text and owner responses. No login, no API key; CSV/JSON/API.

ctrip-trip-reviews-scraper★ 4

Trip.com & Ctrip (携程) hotel reviews scraper on Apify — international + Chinese reviews in one run: sub-ratings, owner responses, reviewer geo, LLM-ready markdown.

expedia-hotel-reviews-scraper★ 4

Expedia hotel reviews scraper on Apify — all 7 Expedia Group brands in one actor: guest ratings, category sub-ratings, review text and owner responses. CSV/JSON/API.

bossJD★ 4

Local crawler and analysis toolkit for AI/Agent technical jobs on BOSS Zhipin.

wechat-article-scraping-skill★ 4

微信公众号文章抓取 → Markdown / 纯文本 / JSON。单文件零依赖 TypeScript Skill,Node 直接跑,无需浏览器或 npm install。WeChat official-account (mp.weixin.qq.com) article scraper: single-file, zero-dependency, agent-skill ready.

BOB-Youtube★ 4

BOB YouTube: Docker-first YouTube ingestion runtime for developers, products, bulk workflows, and AI agents. Powered by yt-dlp, wrapped as a clean CLI, OpenAPI service, /data corpus contract, and release-proof workflow.

ytubefetch★ 4

Free Windows desktop app for batch-downloading YouTube subtitles, audio, and video from playlists and channels, with clean plain-text output for AI summarization.

truffles★ 3

Tools for data extraction, QA and automation from the web with LLMs

purify★ 3

Single-binary web scraper for AI agents. Headless Chrome + Readability → clean Markdown. Up to 99% token savings. Zero dependencies. Built-in MCP server.

openai-sdk-with-web-unlocker★ 3

Integrating OpenAI Agents SDK with Bright Data Web Unlocker, enabling AI agents to access, extract, and process structured data from protected web pages

otodom_scraper_and_information_retrieval★ 3

Otodom scraper and information retrieval

AI-Web-Reader-Chatbot-Extention★ 3

AI-Web-Reader-Chatbot-Extension is a smart Chrome extension that lets you interact with any webpage using AI. It scrapes and analyzes the content of the current website and answers your questions in real time. Powered by advanced language models, it delivers accurate and concise answers based only on the page content.

llm-chat-scraper-skill★ 3

OpenClaw skill for scraping ChatGPT, Gemini, Perplexity and Grok responses. Built for AI agents and GEO monitoring.

cloudscraper.js★ 3

Agent-ready Node.js/TypeScript client for Cloudflare-protected sites: reusable hot sessions, an MCP server + LangChain tool, and clean Markdown for LLMs. Powered by the Python cloudscraper engine.

DocsForge★ 3

docsforge — Crawler and Adaptive Harvest for Technical Documents like Code Language , API , Technologies. Search -> Identify URL -> Crawl -> Harvest Documentation

thordata-firecrawl★ 2

Thordata Firecrawl – Firecrawl-compatible web crawling & scraping API built on Thordata, turning any website into AI-ready Markdown/JSON/HTML/screenshots.

agentfetch★ 2

Open-source web retrieval & research agent built for AI agents. Works with LangChain, LlamaIndex, CrewAI, OpenAI, MCP, and any REST agent. Supports scrape, search, crawl, map, extract, and research.

CRAWL-GPT-CHAT★ 2

docker region:us

FlexiWeb_Stream_Scraper★ 2

An LLM web-chat scraping tool that mimics API functionality, supporting a wide range of Large Language Models. 一个模仿API功能的LLM网页聊天文本抓取工具,支持多种大语言模型。

Crawloop★ 2

Self-healing web crawler — compile your LLM scraper into free, deterministic code that auto-heals when a site's layout changes.

Agentic-Blog★ 2

Read public Naver Blog content with a small, agent-friendly CLI.

hermes-sift★ 2

Resilient web extraction for Hermes Agent. Automatically switches between extraction providers and validates each result to catch incomplete or blocked pages.

floppydata

FloppyData review: compare features, pricing, use cases, access model, and alternatives for this Model Serving agent in 2026. FloppyData offers high-speed residential and mobile proxy services for web scraping and data protect

handinger

Handinger review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. 20x cheaper API to extract website content for AI Agents

datavist

Datavist review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Agentic Web Data Extraction. Pay-as-you-go. No coding required. Prompts are optional.

cliprun

Cliprun review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Instantly run Python code online with just a right-click.

Jsonify

Get key info from millions of online sources with AI agents.

Firecrawl

Get clean, AI-ready data from any website, fast and reliably.

Anon

Automate data extraction from any website, even without APIs.

Kadoa

Automate web data collection with AI, no coding needed.

ScrapeGraphAI

Instantly turn any website into a data API with AI.

Scrape.do

Get clean, LLM-ready web data easily, bypassing tricky websites.

@mendable/n8n-nodes-firecrawl

Official Firecrawl nodes for n8n - scrape, crawl, map, search, and extract data from websites. Supports AI Agent tool usage.

isbot

🤖/👨‍🦰 Recognise bots/crawlers/spiders using the user agent string.

More in Data, search & knowledge