Web scraping & crawling
676 agents · most starred first
Web scraping & crawling, page 4 of 7
Optimize web pages for LLM agent interaction and understanding
Fastest, most accurate & effecient user agent string parser, uses Browserscope's research for parsing
Detect Browser, Engine, OS, CPU, and Device type/model from User-Agent data. Supports browser & node.js environment
A specification compliant robots.txt parser with wildcard (*) matching support.
Fast & effecient user agent string parser. Uses uap-core (Browserscope) data for parsing. Drop-in replacement for useragent
This package is for getting latest agents and randomize it
JavaScript SDK for Firecrawl, the context API for AI agents: search, scrape, parse, crawl, map, and interact with the web, run Agent and Monitor jobs, and search scientific papers (PubMed, bioRxiv, medRxiv, arXiv)
Tavily AI SDK tools - Search, Extract, Crawl, and Map
Firecrawl tools for Vercel AI SDK - Web scraping and search capabilities for AI applications
JavaScript SDK for Firecrawl, the context API for AI agents: search, scrape, parse, crawl, map, and interact with the web, run Agent and Monitor jobs, and search scientific papers (PubMed, bioRxiv, medRxiv, arXiv)
Agentic SDK for Firecrawl.
Agentic SDK for Diffbot.
Agentic SDK for Jina AI.
A universal web crawler for LLM frameworks, compatible with Tavily API for extract and search, and integrates with Hono.js and Model Context Protocol (MCP).
A Google Custom Search and Google Sheets integration for querying and logging results
Distributed Python web crawling framework
CLI tool to scrape posts, comments, and reactions from private Facebook groups
A Python library to extract structured data from web pages using LLMs.
RAGScraper is a Python library designed for efficient and intelligent scraping of web documentation and content. Tailored for Retrieval-Augmented Generation systems, RAGScraper extracts and preprocesses text into structured, machine-learning-ready formats. It emphasizes precision, context preservation, and ease of integration with RAG models, making it an ideal tool for developers looking to enhance AI-driven applications with rich, web-sourced knowledge.
Ragno Passive URL Crawler - Python Module
A package that makes interacting with Scratch API easier
A middleware to change user-agent in request for Scrapy
A package for testing user agents on specific websites
Generates safe and dummy usable user agent and xms user agent
Use for randomly get header
Generated User Agent and Scrap Proxy [http, socka4, socka5](Auto update)
Binary web cartography for AI agents
Tiny Fish AgentQL Python Client
Agent-based web scraping with LLM integration
Give any AI agent the ability to search, crawl, and extract the web.
A simple package for scraping websites and generating responses using Google's Generative AI.
An asynchronous Python API client for LANraragi, written with aiohttp and pydantic for type validation.
AllMusic review scraper
A Fixed and Updated Alternative for fake-useragent
fake pc or app browser useragent, anti useragent, and other awesome tools
Apify integration for Haystack: load Actor results as Documents, search the web, and give Agents access to Apify Actors.
User Agent Generator creates realistic user agent strings for desktop, mobile, and tablet devices. Each request outputs the complete string with parsed browser and operating system names and versions, while paid plans add engine and CPU details.
User Agent Parser parses user agent strings to extract browser, operating system, and hardware details. It identifies device types, CPU architecture, and bot traffic, so you can run user agent detection on every incoming HTTP request.
AutoGen tools for SnapRender Screenshot API: capture websites as PNG, JPEG, WebP, or PDF
A web scraping and data extraction tool.
YouTube video search tools for AI agent pipelines
get a browser user_agent by random
Browserbase Haystack Fetcher
由于国内网络限制无法使用 fake_useragent 库而制作的替代品
A library for cleaning HTML content by removing specified tags and attributes.
A Python package for scraping CNKI papers.
Thin client for the Cortex web cartography runtime — map, compile, and query websites from Python
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders.
DEPRECATED - renamed to crewai-lastsearch. https://lastsearch.ai/migrate
A CrewAI tool for smart web scraping via the Browserless /smart-scrape API.
CRW web scraping tools for CrewAI — scrape, crawl, and map websites
CrewAI tools for web scraping and research using Olostep API
CrewAI tool for loading web pages using Tzafon's headless browser infrastructure
Host your own ddragon mirror for League of Legends static data and images, relying on dragontail
Parses user agent string built with the device_user-agent flutter package.
User agent parser
django-smartagent is the fastest and most complete user agent parser
A django package that allows easy identification of visitors' browser, operating system and device information (mobile phone, tablet or has touch capabilities).
A django package that allows easy identification of visitors' browser, operating system and device information (mobile phone, tablet or has touch capabilities).
Detect and handle desktop vs mobile user agents with django-annoying
A django package that allows easy identification of visitors' browser, operating system and device information (mobile phone, tablet or has touch capabilities).
Extract the main article content (and optionally comments) from a web page
A python libary for dragonball-germany.de
Download from tik tok and youtube and facebook and other soon
random useragent without network
Generate fake user agents on the fly
A professional random User-Agent generator for web scraping.
Up-to-date simple useragent faker with real world database
Up to date simple useragent faker with real world database
fake_useragent.
Up to date simple useragent faker with real world database
Fake User-Agent Generator with Gray Randomization
Simple package with fake user agents for your requests.
Randomly generate a valid useragent for faking a browser.
Haystack integration for Firecrawl
A utility for finding feeds on websites
The official Python client for Forager.
Up-to-date simple user-agent faker with real world database
Select and reverse-Markdown (html2text) web page fragments.
A wrapper for BeautifulSoup4 that restores the ability to work with HTML fragments
Simple User Agent Generator
Asynchronous Python client for getting garage occupancy in Burgos
A library to identify devices (phones, tablets) and their capabilities by parsing (browser/HTTP) user agent strings
This package contains implementation of a web scraper with Gemini Pro integrated into it.
Random Useragent
Give CrewAI agents native Geonode web scraping tools to extract pages, search the web, discover URLs and manage crawl or batch jobs.
Give LangChain agents native Geonode web scraping tools to extract pages, search the web, discover URLs and manage crawl or batch jobs.
A package to fetch get-user-agent strings
This library is used to make it easier for developers or users to get random user agents
Get a random, popular, commonly-used user agent string. Random/fake/spoof user agents.
A package for gather the online data using GPT.
GPT-Scraper is an autonomous, LLM-based agent that generates code to extract structured information from web pages.
A web crawler for GPTs to build knowledge bases
Unofficial GuteFrage api. Check https://github.com/DAMcraft/gutefrage/wiki on how to use it
Bright Data integration for Haystack - web scraping, SERP API, and data extraction from 45+ websites
CLI para baixar hqs no site.
Extracts OS Browser etc information from http user agent string
Client interface for Scrapinghub HubStorage