Data, search & knowledge

Web scraping & crawling

676 agents · most starred first

Web scraping & crawling, page 7 of 7

llama-index-readers-wellmarked

Official LlamaIndex reader for the WellMarked API — load any URL as clean Markdown.

llamaindex-proxyhat

LlamaIndex web reader that loads URLs through ProxyHat residential proxies — rotation, geo-targeting, sticky sessions.

scrapeunblocker-haystack

Haystack integration for ScrapeUnblocker - scrape pages behind anti-bot protections

user-agents-ng

A maintained successor to user-agents: identify devices (phones, tablets) and their capabilities by parsing browser user agent strings.

ytagent-cli

An agentic YouTube downloader for cloud-based AI agents. Multi-method fallback chain. No cookies, no browser, no LLM at runtime.

@danypops/web-spider

AI-agent-friendly web spider: structured output, RAG-ready chunks, graph-traversable links

llama-index-tools-olostep

llama-index tools olostep integration

writ-agent

Official Python SDK for Writ (usewrit.app): call your website APIs on Writ Cloud and Writ Desktop: workflows, monitors, scrape, map and crawl from one client

@x12i/web-agentic-api

HTTP host for web-agentic: fetch, classify, parse, ingest, and Serper search. Embeddable Fastify app or runnable bin (zone web-agentic-tools → :7080).

crewai-transcriptapi

CrewAI tools for TranscriptAPI: fetch YouTube transcripts, search YouTube, and pull video metadata.

transcriptfetch

TranscriptFetch review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. API for extracting transcripts from video and podcast platforms

scrape402-langchain

An x402-gated web scraper tool for LangChain agents, powered by Algorand.

langchain-context

LangChain tools for Context.dev web data, brand intelligence, monitors, and batches

langchain-hydrafetch

LangChain integration for Hydrafetch. Load, search and extract clean web data for agents and RAG.

llama-index-readers-hydrafetch

LlamaIndex reader for Hydrafetch. Load clean, LLM-ready web data into an index.

crewai-webmetadata-extractor

CrewAI tools for the Web Metadata, OpenGraph & Contact Extractor API — give your crew URL-to-structured-data extraction, AI-ready Markdown, contact discovery, and SEO auditing.

gemini-scraper

Gemini scraper and Gemini API endpoint client built on ScrapingBee. Send prompts to Google Gemini, get text, markdown and citations as JSON.

langchain-gluedly

LangChain document loader for Gluedly scrape snapshots

langchain-quanticdata

LangChain integration for QuanticData — document loaders, a web retriever and an agent toolkit for scraping, SERP search, crawling and 74 ready-made collectors through residential proxies.

langchain-webmetadata-extractor

LangChain tools for the Web Metadata, OpenGraph & Contact Extractor API — give your agent URL-to-structured-data extraction, AI-ready Markdown, contact discovery, and SEO auditing.

llama-index-readers-gluedly

LlamaIndex reader for Gluedly scrape snapshots

llamaindex-webmetadata-extractor

LlamaIndex reader and tools for the Web Metadata, OpenGraph & Contact Extractor API — ingest any URL as a clean-Markdown Document for RAG, plus agent tools for contacts, tech stack, and SEO auditing.

langchain-hackernews

LangChain retriever integration for Hacker News — search stories, fetch the front page, and retrieve full comment threads without API keys.

langchain-opticparse

Official LangChain 1.x & CrewAI Integration for OpticParse Vision Web Scraping & PhishVision Threat Detection (Includes 200 Free Trial Requests & Zero-Friction In-IDE Refills)

langchain-playstore

LangChain retriever integration for Google Play Store — fetch app details and retrieve user reviews.

llama-index-tools-opticparse

Official LlamaIndex BaseToolSpec Integration for OpticParse Vision Web Scraping & PhishVision Threat Detection

anyapi-haystack

Haystack components and tools for AnyAPI: one key, hundreds of data and scraping APIs, billed per request in USD.

gologin

Gologin Cloud Browser review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Cloud browser API that maintains persistent logged-in sessions.

@ihentrel/x402-scraper-langchain

Autonomous x402 web scraper tool for AI agents with automatic Base mainnet USDC micro-settlement

canonicalwebteam.store-llm

Flask extension serving a store's pages as Markdown at /page.md and listing them in llms.txt and llms-full.txt.

fetta-langchain

LangChain document loader that uses fetta - tool that fetches web pages for assistants

langchain-xdataapi

LangChain tools for xdataapi.io: read-only public X (Twitter) profiles, tweets, threads, followers and search.

llama-index-readers-enconvert

EnConvert reader for LlamaIndex: perceive URLs or ingest a whole site into Documents, every read scored.

whatsmynameapps

WhatsMyNameApps review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Rated 5/5 from 1 verified review. Search for usernames across thousands of social media sites

crewai-getyoutubetranscript

YouTube transcript tools for CrewAI agents: get YouTube video transcripts with timestamps, search YouTube, and list channel videos, via the GetYouTubeTranscript API.

flashdata-youtube-cloud-storage

Deliver authorized YouTube media to your cloud storage with the FlashData JSON CLI

langchain-arcmira

YouTube transcript search and entity resolution tools for LangChain and LangGraph.

langchain-comment-scraper

LangChain document loader for Reddit and Hacker News threads exported with Comment Scraper. Reads local JSON files, makes no network requests.

langchain-decodo

LangChain integration for the Decodo web scraping API

langchain-geekflare

LangChain tools for Geekflare: web search, scraping, screenshots, PDFs, brand data, and website diagnostics.

langchain-getyoutubetranscript

YouTube transcript tools and document loader for LangChain: get YouTube video transcripts with timestamps and search YouTube from agents and RAG pipelines, via the GetYouTubeTranscript API.

langchain-mesharc

LangChain integration for MeshArc: web pages as Documents, and web search and page reading as agent tools.

langchain-printfriendly

LangChain integration for the PrintFriendly API: load any web page as clean, ad-free Markdown.

langchain-proxmint

LangChain tool and document loader that read web pages through Proxmint residential, datacenter and mobile proxies, with country, city and ASN targeting and sticky sessions.

langchain-scrapely

LangChain integration for Scrape.ly: run Actors, load their results as documents, and give agents Scrape.ly tools.

langchain-scrapewise

LangChain tools for the ScrapeWise web-scraping and price-monitoring API.

langchain-urlpipe

LangChain tools, toolkit and document loader for URLpipe: read any web page as Markdown, or get its metadata, data in your own fields, its technologies, console errors or a Lighthouse audit, rendered in real Chrome.

langchain-webextract

LangChain tools for WebExtract: extract_url (Markdown + product/price/tables JSON) and page_markdown. Pay per call with x402, prepaid key, or free tier.

llama-index-readers-anakin

LlamaIndex reader for Anakin: load web pages, crawls, search results and site maps as Documents

llama-index-readers-contextfetch

llama-index readers contextfetch integration

llama-index-readers-decodo

Decodo web scraping API reader for LlamaIndex — load web pages and search results as Documents.

llama-index-readers-mesharc

LlamaIndex reader for MeshArc: web pages, crawls, sitemaps and change records as Documents.

llama-index-readers-printfriendly

llama-index readers printfriendly integration: load any web page as clean, ad-free Markdown

llama-index-readers-scrapely

LlamaIndex readers for Scrape.ly: run Actors or read datasets and get Documents back.

llama-index-readers-urlpipe

LlamaIndex reader for URLpipe: load web pages as clean Markdown, rendered HTML or a summary, rendered in real Chrome.

llama-index-tools-anakin

LlamaIndex tools for Anakin: web scraping, search, deep research, Wire site actions, monitors and browser tasks

llama-index-tools-clawfetch

llama-index tools clawfetch integration — pay-per-call web intelligence for AI agents (x402, USDC on Base)

llama-index-tools-contextfetch

llama-index tools contextfetch integration

llama-index-tools-decodo

Decodo web scraping API tool spec for LlamaIndex agents.

llama-index-tools-geekflare

LlamaIndex tools for Geekflare: web search, Google SERP, scraping, screenshots, PDFs, brand data, and website diagnostics.

llama-index-tools-tanod

LlamaIndex tool spec for Tanod: web search, page/PDF/document to Markdown, screenshots, OCR, DNS and email checks, phishing URL check, address risk, contract and package scans for AI agents. Free daily tier, then x402 USDC.

ohamlab-agent

Web research and extraction toolkit with MCP and optional autonomous LLM agents

proxmint-haystack

Haystack component that fetches web pages through Proxmint residential, datacenter and mobile proxies, with country, city and ASN targeting and sticky sessions.

decodo-crewai

Decodo Web Data API tools for CrewAI agents.

crewai-alterlab

CrewAI integration for AlterLab web scraping API

langchain-alterlab

LangChain integration for AlterLab web scraping API

llama-index-readers-alterlab

LlamaIndex data reader for AlterLab web scraping API

llama-index-readers-x402-cleanweb

LlamaIndex official data reader integration for x402 CleanWeb Studio (87% token reduction, multimodal alt-text, and prompt injection defense)

NanoParse

Clean Markdown from any URL, with Litmus trust signals. 20 deterministic signals (source authority, freshness, correction tracking, paywall & truncation, syndication, content-type, reading time and more) so an agent knows what it actually read. $0.005 per page in USDC on Base via x402; failed reads are never charged. No account: 3 anonymous free tries per network, or 10 free parses per month with a signed wallet. Hosted MCP server: https://nanoparse.app/mcp

Outcrawl

Web infrastructure for AI agents. Scrape any URL to clean markdown or JSON, crawl a whole site, search the web with the results already read, drive real browsers in the cloud over CDP, or hand a task to a browser agent that finishes it. One credit balance, MCP server and REST API, free plan with 1,000 credits a month.

langchain-pagesift

LangChain document loader for pagesift: web pages, sites and local files as Markdown, no API key.

llama-index-readers-pagesift

LlamaIndex reader for pagesift: web pages, sites and local files as Markdown, no API key.

paragraphic

An Obsidian-style graph of how a static site's pages link to each other, read from its rendered HTML

passageway-connector-llmstxt

passageway connector for the llms-full.txt of a list of websites

Wrenda

Wrenda is a reverse proxy that sits in front of your site and detects who is requesting each page. AI crawlers get enriched markdown with schema and FAQs. Search bots get fully pre-rendered HTML. Humans get your normal page.

XCrawl

XCrawl is a web data API for AI builders: scrape, crawl, map and search the web and get clean JSON or Markdown back, with proxies and JS rendering handled for you.

More in Data, search & knowledge