Web scraping & crawling
676 agents · most starred first
Web scraping & crawling, page 7 of 7
Official LlamaIndex reader for the WellMarked API — load any URL as clean Markdown.
LlamaIndex web reader that loads URLs through ProxyHat residential proxies — rotation, geo-targeting, sticky sessions.
Haystack integration for ScrapeUnblocker - scrape pages behind anti-bot protections
A maintained successor to user-agents: identify devices (phones, tablets) and their capabilities by parsing browser user agent strings.
An agentic YouTube downloader for cloud-based AI agents. Multi-method fallback chain. No cookies, no browser, no LLM at runtime.
AI-agent-friendly web spider: structured output, RAG-ready chunks, graph-traversable links
llama-index tools olostep integration
Official Python SDK for Writ (usewrit.app): call your website APIs on Writ Cloud and Writ Desktop: workflows, monitors, scrape, map and crawl from one client
HTTP host for web-agentic: fetch, classify, parse, ingest, and Serper search. Embeddable Fastify app or runnable bin (zone web-agentic-tools → :7080).
CrewAI tools for TranscriptAPI: fetch YouTube transcripts, search YouTube, and pull video metadata.
TranscriptFetch review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. API for extracting transcripts from video and podcast platforms
An x402-gated web scraper tool for LangChain agents, powered by Algorand.
LangChain tools for Context.dev web data, brand intelligence, monitors, and batches
LangChain integration for Hydrafetch. Load, search and extract clean web data for agents and RAG.
LlamaIndex reader for Hydrafetch. Load clean, LLM-ready web data into an index.
CrewAI tools for the Web Metadata, OpenGraph & Contact Extractor API — give your crew URL-to-structured-data extraction, AI-ready Markdown, contact discovery, and SEO auditing.
Gemini scraper and Gemini API endpoint client built on ScrapingBee. Send prompts to Google Gemini, get text, markdown and citations as JSON.
LangChain document loader for Gluedly scrape snapshots
LangChain integration for QuanticData — document loaders, a web retriever and an agent toolkit for scraping, SERP search, crawling and 74 ready-made collectors through residential proxies.
LangChain tools for the Web Metadata, OpenGraph & Contact Extractor API — give your agent URL-to-structured-data extraction, AI-ready Markdown, contact discovery, and SEO auditing.
LlamaIndex reader for Gluedly scrape snapshots
LlamaIndex reader and tools for the Web Metadata, OpenGraph & Contact Extractor API — ingest any URL as a clean-Markdown Document for RAG, plus agent tools for contacts, tech stack, and SEO auditing.
LangChain retriever integration for Hacker News — search stories, fetch the front page, and retrieve full comment threads without API keys.
Official LangChain 1.x & CrewAI Integration for OpticParse Vision Web Scraping & PhishVision Threat Detection (Includes 200 Free Trial Requests & Zero-Friction In-IDE Refills)
LangChain retriever integration for Google Play Store — fetch app details and retrieve user reviews.
Official LlamaIndex BaseToolSpec Integration for OpticParse Vision Web Scraping & PhishVision Threat Detection
Haystack components and tools for AnyAPI: one key, hundreds of data and scraping APIs, billed per request in USD.
Gologin Cloud Browser review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Cloud browser API that maintains persistent logged-in sessions.
Autonomous x402 web scraper tool for AI agents with automatic Base mainnet USDC micro-settlement
Flask extension serving a store's pages as Markdown at /page.md and listing them in llms.txt and llms-full.txt.
LangChain document loader that uses fetta - tool that fetches web pages for assistants
LangChain tools for xdataapi.io: read-only public X (Twitter) profiles, tweets, threads, followers and search.
EnConvert reader for LlamaIndex: perceive URLs or ingest a whole site into Documents, every read scored.
WhatsMyNameApps review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Rated 5/5 from 1 verified review. Search for usernames across thousands of social media sites
YouTube transcript tools for CrewAI agents: get YouTube video transcripts with timestamps, search YouTube, and list channel videos, via the GetYouTubeTranscript API.
Deliver authorized YouTube media to your cloud storage with the FlashData JSON CLI
YouTube transcript search and entity resolution tools for LangChain and LangGraph.
LangChain document loader for Reddit and Hacker News threads exported with Comment Scraper. Reads local JSON files, makes no network requests.
LangChain integration for the Decodo web scraping API
LangChain tools for Geekflare: web search, scraping, screenshots, PDFs, brand data, and website diagnostics.
YouTube transcript tools and document loader for LangChain: get YouTube video transcripts with timestamps and search YouTube from agents and RAG pipelines, via the GetYouTubeTranscript API.
LangChain integration for MeshArc: web pages as Documents, and web search and page reading as agent tools.
LangChain integration for the PrintFriendly API: load any web page as clean, ad-free Markdown.
LangChain tool and document loader that read web pages through Proxmint residential, datacenter and mobile proxies, with country, city and ASN targeting and sticky sessions.
LangChain integration for Scrape.ly: run Actors, load their results as documents, and give agents Scrape.ly tools.
LangChain tools for the ScrapeWise web-scraping and price-monitoring API.
LangChain tools, toolkit and document loader for URLpipe: read any web page as Markdown, or get its metadata, data in your own fields, its technologies, console errors or a Lighthouse audit, rendered in real Chrome.
LangChain tools for WebExtract: extract_url (Markdown + product/price/tables JSON) and page_markdown. Pay per call with x402, prepaid key, or free tier.
LlamaIndex reader for Anakin: load web pages, crawls, search results and site maps as Documents
llama-index readers contextfetch integration
Decodo web scraping API reader for LlamaIndex — load web pages and search results as Documents.
LlamaIndex reader for MeshArc: web pages, crawls, sitemaps and change records as Documents.
llama-index readers printfriendly integration: load any web page as clean, ad-free Markdown
LlamaIndex readers for Scrape.ly: run Actors or read datasets and get Documents back.
LlamaIndex reader for URLpipe: load web pages as clean Markdown, rendered HTML or a summary, rendered in real Chrome.
LlamaIndex tools for Anakin: web scraping, search, deep research, Wire site actions, monitors and browser tasks
llama-index tools clawfetch integration — pay-per-call web intelligence for AI agents (x402, USDC on Base)
llama-index tools contextfetch integration
Decodo web scraping API tool spec for LlamaIndex agents.
LlamaIndex tools for Geekflare: web search, Google SERP, scraping, screenshots, PDFs, brand data, and website diagnostics.
LlamaIndex tool spec for Tanod: web search, page/PDF/document to Markdown, screenshots, OCR, DNS and email checks, phishing URL check, address risk, contract and package scans for AI agents. Free daily tier, then x402 USDC.
Web research and extraction toolkit with MCP and optional autonomous LLM agents
Haystack component that fetches web pages through Proxmint residential, datacenter and mobile proxies, with country, city and ASN targeting and sticky sessions.
Decodo Web Data API tools for CrewAI agents.
CrewAI integration for AlterLab web scraping API
LangChain integration for AlterLab web scraping API
LlamaIndex data reader for AlterLab web scraping API
LlamaIndex official data reader integration for x402 CleanWeb Studio (87% token reduction, multimodal alt-text, and prompt injection defense)
Clean Markdown from any URL, with Litmus trust signals. 20 deterministic signals (source authority, freshness, correction tracking, paywall & truncation, syndication, content-type, reading time and more) so an agent knows what it actually read. $0.005 per page in USDC on Base via x402; failed reads are never charged. No account: 3 anonymous free tries per network, or 10 free parses per month with a signed wallet. Hosted MCP server: https://nanoparse.app/mcp
Web infrastructure for AI agents. Scrape any URL to clean markdown or JSON, crawl a whole site, search the web with the results already read, drive real browsers in the cloud over CDP, or hand a task to a browser agent that finishes it. One credit balance, MCP server and REST API, free plan with 1,000 credits a month.
LangChain document loader for pagesift: web pages, sites and local files as Markdown, no API key.
LlamaIndex reader for pagesift: web pages, sites and local files as Markdown, no API key.
An Obsidian-style graph of how a static site's pages link to each other, read from its rendered HTML
passageway connector for the llms-full.txt of a list of websites
Wrenda is a reverse proxy that sits in front of your site and detects who is requesting each page. AI crawlers get enriched markdown with schema and FAQs. Search bots get fully pre-rendered HTML. Humans get your normal page.
XCrawl is a web data API for AI builders: scrape, crawl, map and search the web and get clean JSON or Markdown back, with proxies and JS rendering handled for you.