Web scraping & crawling
676 agents · most starred first
Web scraping & crawling, page 6 of 7
LLM integration for Scrapy
A Scrapy extension for data extraction using LangChain
Scrapy extension to store info in storage service
Scrapy Middleware to set a random User-Agent for every Request.
A random user-agent for all your needs
Upload scrapy logs to S3
Redis-based components for Scrapy.
Automatically pick an User-Agent for every request
Always get best user-agent
An async wrapper for the ShadowDragon API
Seamlessly fetch authentic and current user agents with ease
A library to identify devices (phones, tablets) and their capabilities by parsing browser user agent strings.
smappdragon is a set of tools for working with twitter data
A package to get random user agents based filters provided by user
A distributed spider framework
easy local file transfer with curvecp
Weather forecast data
A small crawler to scrape data from swranking.com and store in in the local db of the vm.
Web Scraping Tool for Swarmauri
A Python library for accessing Google Trends data
A Python library for accessing Google Trends data
Free, open-source Python library for Google Trends data: trending now, keyword interest over time, related queries, interest by region, 2-5-keyword comparison and YouTube/News/Images/Shopping interest. A maintained pytrends alternative with a pytrends-compatible TrendReq, a CLI and an MCP server for AI agents.
Python3 client for accessing tv show information from www.tvrage.com
The easiest way to crawl a website and produce LLM ready markdown files
Library to build content for User-Agent HTTP header
user_agent
user_agent
随机ua
Django User Agent is a tracker for visitors on the website through the API
Python 3 library to parse User-Agent strings and detect browser, OS and device details
听海内部爬虫核心类库
scrape user agents
aaaaaaa
Simple user agent generator and interpreter tools
generate random user agent
A library to identify devices (phones, tablets) and their capabilities by parsing browser user agent strings.
A library to identify devices (phones, tablets) and their capabilities by parsing (browser/HTTP) user agent strings
Automatically Generate User agents
UNKNOWN
User-Agent changing library for Python.
Generates random but realistic user agents on a command line (or via API)
Random generation of Uesr agent for PC and mobile
An ease-of-access and non-intrusive Python web scraper using beautifulsoup to easily fetch and filter through the latest user agents found in useragents.me
Python port of the Java user-agent-utils lib
Random generation of User agent for PC and mobile
A simple user agent random
Random user agent
user agent random
LLM <3 WARC files
WaterCrawl OpenAI Plugin
Web fragments
Web Structured Data Extraction Agent
A tool to scrape web content into clean Markdown for LLMs.
LangChain integration for WebCrawlerAPI
Official Python SDK for the WellMarked API — convert any URL to clean Markdown.
Self updating package for generating random user agent strings.
Apify review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Largest ecosystem for building, deploying, and publishing scraping and automation tools.
Effortlessly extract web data at scale. No code, no worries.
Tile source to dynamically create tiles if its missing in the storage
Scrape.do review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Extract LLM-Ready Data From Any Website
Easily get public data from any website, bypassing blocks with AI.
Automate data gathering from websites and documents with AI.
Extract web data by chatting with AI. No coding needed.
A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format.
Web content extraction and parsing library with HTML-to-Markdown conversion
Apify extracts live web data for training and running AI.
Scrape websites and use AI to make sense of your data.
Command-line interface for Firecrawl. Scrape, crawl, and extract data from any website directly from your terminal.
Node SDK for Scrapeless AI
Add crawling capability to pydantic ai agent
An integration package connecting Firecrawl and LangChain
Crawl any website and generate llms.txt - the AI-ready site map standard.
LangChain tool and document loader for Lyrenth: read URLs as clean AIDocuments.
Curated, sourced list of AI crawler / training bot user agents, plus a small CLI to test whether a URL is reachable to each bot.
A LangChain integration tool that provides reliable web scraping capabilities at any scale using ZenRows' Universal Scraper API, extended with Adaptive Stealth Mode
Self-hosted web scraping and Markdown extraction for AI agents.
Free, open-source, async-first web scraping toolkit for LLM agents
LangChain tool for Skim — clean web reader for AI agents. Card plan with API key (free tier) or x402 wallet pay-per-call ($0.002 USDC on Base).
LlamaIndex reader for Skim — clean web reader for AI agents. Card plan with API key (free tier) or x402 wallet pay-per-call ($0.002 USDC on Base).
Haystack integration for Skim — clean web reader for AI agents. Card plan with API key (free tier) or x402 wallet pay-per-call ($0.002 USDC on Base).
LangChain tools for the GhostCrawl web data API.
Scraperify review: compare features, pricing, use cases, access model, and alternatives for this Web Scraping agent in 2026. Chrome extensions for one-click data export from websites without coding
Characterize the clients hitting a web site by analyzing its access logs.
LangChain tools for Novada — web search, scrape, crawl, extract, research & proxy — derived LIVE from the Novada MCP server (zero contract drift).
LangChain tools for the Synoppy web-data API — read, search, crawl, map, extract, classify, enrich, images, and screenshot the live web from your agent.
An integration package connecting TheCrawler and LangChain
Random User-Agent string generator with embedded browser templates
Server-side SDK for Agentic Scraper MCP/API provider monetization.
Read public Naver Blog content with a small, agent-friendly CLI.
Read your own logged-in Facebook — timelines, feed, comments, search, groups — into clean JSON, over the same GraphQL endpoint your browser uses.
Agentic Reddit — anonymous browser-backed, read-only Reddit retrieval.
Read-only Threads retrieval through authenticated, single-target CLI primitives
Read-only scraping of logged-in X/Twitter data via a harvest-then-replay hybrid: a stealth-browser login harvests the session once, then reads are replayed over httpx. No client-transaction-id replay.
A modern Python wrapper for the theCrag Logbook API.
Universal web collection engine — extract, monitor, and search any website through Cloudflare
LangChain document loader and tool that read the web through ProxyHat residential proxies — rotation, geo-targeting, sticky sessions.
Tabstack tools for LangChain. Schema-enforced web extraction and research, backed by the official Tabstack SDK.
Official LangChain integration for the WellMarked API — load any URL as clean Markdown.