Data, search & knowledge

Web scraping & crawling

676 agents · most starred first

Web scraping & crawling, page 6 of 7

scrapy-llm

LLM integration for Scrapy

scrapy-llm-loader

A Scrapy extension for data extraction using LangChain

scrapy-pagestorage

Scrapy extension to store info in storage service

scrapy-random-useragent

Scrapy Middleware to set a random User-Agent for every Request.

scrapy-random-useragent-pro

A random user-agent for all your needs

scrapy-s3logstorage

Upload scrapy logs to S3

scrapy-swarm

Redis-based components for Scrapy.

scrapy-user-agents

Automatically pick an User-Agent for every request

shadow-useragent

Always get best user-agent

shadowdragon

An async wrapper for the ShadowDragon API

simple-useragent

Seamlessly fetch authentic and current user agents with ease

slipmatio-user-agents

A library to identify devices (phones, tablets) and their capabilities by parsing browser user agent strings.

smappdragon

smappdragon is a set of tools for working with twitter data

solotop-random-user-agent

A package to get random user agents based filters provided by user

spiderswarm

A distributed spider framework

spiral-swarm

easy local file transfer with curvecp

sunnyday-raghu

Weather forecast data

swagenttools

A small crawler to scrape data from swranking.com and store in in the local db of the vm.

swarmauri_tool_webscraping

Web Scraping Tool for Swarmauri

trendspy

A Python library for accessing Google Trends data

trendspy-lite

A Python library for accessing Google Trends data

trendspyg

Free, open-source Python library for Google Trends data: trending now, keyword interest over time, related queries, interest by region, 2-5-keyword comparison and YouTube/News/Images/Shopping interest. A maintained pytrends alternative with a pytrends-compatible TrendReq, a CLI and an MCP server for AI agents.

tvrage3

Python3 client for accessing tv show information from www.tvrage.com

url2llm

The easiest way to crawl a website and produce LLM ready markdown files

user-agent

Library to build content for User-Agent HTTP header

user-agent-collect

user_agent

user-agent-data

user_agent

user-agent-gather

随机ua

user-agent-middleware

Django User Agent is a tracker for visitors on the website through the API

user-agent-parser

Python 3 library to parse User-Agent strings and detect browser, OS and device details

user-agent-random

听海内部爬虫核心类库

user-agent-scraper

scrape user agents

user-agent-th

aaaaaaa

user-agent-tools

Simple user agent generator and interpreter tools

user-agent2

generate random user agent

user-agents

A library to identify devices (phones, tablets) and their capabilities by parsing browser user agent strings.

user-agents-next

A library to identify devices (phones, tablets) and their capabilities by parsing (browser/HTTP) user agent strings

userAgentRandomizer

Automatically Generate User agents

useragent

UNKNOWN

useragent-changer

User-Agent changing library for Python.

useragent-classifier
useragent-picker-cli

Generates random but realistic user agents on a command line (or via API)

useragent-xxc

Random generation of Uesr agent for PC and mobile

useragents-me-scraper

An ease-of-access and non-intrusive Python web scraper using beautifulsoup to easily fetch and filter through the latest user agents found in useragents.me

useragentutils

Python port of the Java user-agent-utils lib

useragentxxc

Random generation of User agent for PC and mobile

usr-agent-rnd-rs

A simple user agent random

uxagent

Random user agent

vaUserAgent

user agent random

warag
warcllm

LLM <3 WARC files

watercrawl-openai

WaterCrawl OpenAI Plugin

web-fragments

Web fragments

web2json-agent

Web Structured Data Extraction Agent

web2llm

A tool to scrape web content into clean Markdown for LLMs.

webcrawlerapi-langchain

LangChain integration for WebCrawlerAPI

wellmarked

Official Python SDK for the WellMarked API — convert any URL to clean Markdown.

whosyouragent

Self updating package for generating random user agent strings.

apify

Apify review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Largest ecosystem for building, deploying, and publishing scraping and automation tools.

Reworkd

Effortlessly extract web data at scale. No code, no worries.

kartotherian-autogen

Tile source to dynamically create tiles if its missing in the storage

scrapedo

Scrape.do review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Extract LLM-Ready Data From Any Website

Scrapeless

Easily get public data from any website, bypassing blocks with AI.

Otto

Automate data gathering from websites and documents with AI.

Chat4Data

Extract web data by chatting with AI. No coding needed.

@langgraph-js/crawler-mcp

A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format.

@langgraph-js/web-fetch

Web content extraction and parsing library with HTML-to-Markdown conversion

Apify

Apify extracts live web data for training and running AI.

Simplescraper

Scrape websites and use AI to make sense of your data.

firecrawl-cli

Command-line interface for Firecrawl. Scrape, crawl, and extract data from any website directly from your terminal.

@scrapeless-ai/sdk

Node SDK for Scrapeless AI

pydantic-ai-crawlers

Add crawling capability to pydantic ai agent

langchain-firecrawl

An integration package connecting Firecrawl and LangChain

llms-generator

Crawl any website and generate llms.txt - the AI-ready site map standard.

langchain-lyrenth

LangChain tool and document loader for Lyrenth: read URLs as clean AIDocuments.

@geosuite/ai-crawler-bots

Curated, sourced list of AI crawler / training bot user agents, plus a small CLI to test whether a URL is reachable to each bot.

langchain-zenrows-adaptive

A LangChain integration tool that provides reliable web scraping capabilities at any scale using ZenRows' Universal Scraper API, extended with Adaptive Stealth Mode

agentcrawl-ai

Self-hosted web scraping and Markdown extraction for AI agents.

web4agent

Free, open-source, async-first web scraping toolkit for LLM agents

langchain-skim

LangChain tool for Skim — clean web reader for AI agents. Card plan with API key (free tier) or x402 wallet pay-per-call ($0.002 USDC on Base).

llama-index-readers-skim

LlamaIndex reader for Skim — clean web reader for AI agents. Card plan with API key (free tier) or x402 wallet pay-per-call ($0.002 USDC on Base).

skim-haystack

Haystack integration for Skim — clean web reader for AI agents. Card plan with API key (free tier) or x402 wallet pay-per-call ($0.002 USDC on Base).

ghostcrawl-langchain

LangChain tools for the GhostCrawl web data API.

scraperify

Scraperify review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Chrome extensions for one-click data export from websites without coding

agent-census

Characterize the clients hitting a web site by analyzing its access logs.

langchain-novada

LangChain tools for Novada — web search, scrape, crawl, extract, research & proxy — derived LIVE from the Novada MCP server (zero contract drift).

langchain-synoppy

LangChain tools for the Synoppy web-data API — read, search, crawl, map, extract, classify, enrich, images, and screenshot the live web from your agent.

langchain-thecrawler

An integration package connecting TheCrawler and LangChain

useragentgen

Random User-Agent string generator with embedded browser templates

agentic-scraper-provider-sdk

Server-side SDK for Agentic Scraper MCP/API provider monetization.

agentic-blog

Read public Naver Blog content with a small, agent-friendly CLI.

agentic-facebook

Read your own logged-in Facebook — timelines, feed, comments, search, groups — into clean JSON, over the same GraphQL endpoint your browser uses.

agentic-reddit

Agentic Reddit — anonymous browser-backed, read-only Reddit retrieval.

agentic-threads

Read-only Threads retrieval through authenticated, single-target CLI primitives

agentic-twitter

Read-only scraping of logged-in X/Twitter data via a harvest-then-replay hybrid: a stealth-browser login harvests the session once, then reads are replayed over httpx. No client-transaction-id replay.

cragpy

A modern Python wrapper for the theCrag Logbook API.

harvest-agent

Universal web collection engine — extract, monitor, and search any website through Cloudflare

langchain-proxyhat

LangChain document loader and tool that read the web through ProxyHat residential proxies — rotation, geo-targeting, sticky sessions.

langchain-tabstack

Tabstack tools for LangChain. Schema-enforced web extraction and research, backed by the official Tabstack SDK.

langchain-wellmarked

Official LangChain integration for the WellMarked API — load any URL as clean Markdown.

More in Data, search & knowledge