Data, search & knowledge

Web scraping & crawling

676 agents · most starred first

Web scraping & crawling, page 4 of 7

@stdbl/wao

Optimize web pages for LLM agent interaction and understanding

useragent

Fastest, most accurate & effecient user agent string parser, uses Browserscope's research for parsing

my-ua-parser

Detect Browser, Engine, OS, CPU, and Device type/model from User-Agent data. Supports browser & node.js environment

robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

useragent-ng

Fast & effecient user agent string parser. Uses uap-core (Browserscope) data for parsing. Drop-in replacement for useragent

@tonyrl/rand-user-agent

This package is for getting latest agents and randomize it

@mendable/firecrawl-js

JavaScript SDK for Firecrawl, the context API for AI agents: search, scrape, parse, crawl, map, and interact with the web, run Agent and Monitor jobs, and search scientific papers (PubMed, bioRxiv, medRxiv, arXiv)

@tavily/ai-sdk

Tavily AI SDK tools - Search, Extract, Crawl, and Map

firecrawl-aisdk

Firecrawl tools for Vercel AI SDK - Web scraping and search capabilities for AI applications

firecrawl

JavaScript SDK for Firecrawl, the context API for AI agents: search, scrape, parse, crawl, map, and interact with the web, run Agent and Monitor jobs, and search scientific papers (PubMed, bioRxiv, medRxiv, arXiv)

@agentic/firecrawl

Agentic SDK for Firecrawl.

@agentic/diffbot

Agentic SDK for Diffbot.

@agentic/jina

Agentic SDK for Jina AI.

@langgraph-js/crawler

A universal web crawler for LLM frameworks, compatible with Tavily API for extract and search, and integrates with Hono.js and Model Context Protocol (MCP).

CSEGPTV1

A Google Custom Search and Google Sheets integration for querying and logging results

Dragline

Distributed Python web crawling framework

ForageFacebook

CLI tool to scrape posts, comments, and reactions from private Facebook groups

LLMsWebScraper

A Python library to extract structured data from web pages using LLMs.

RAGScraper

RAGScraper is a Python library designed for efficient and intelligent scraping of web documentation and content. Tailored for Retrieval-Augmented Generation systems, RAGScraper extracts and preprocesses text into structured, machine-learning-ready formats. It emphasizes precision, context preservation, and ease of integration with RAG models, making it an ideal tool for developers looking to enhance AI-driven applications with rich, web-sourced knowledge.

Ragno

Ragno Passive URL Crawler - Python Module

ScraGet

A package that makes interacting with Scratch API easier

Scrapy-UserAgents

A middleware to change user-agent in request for Scrapy

UserAgentFilter

A package for testing user agents on specific websites

UserAgentReplica

Generates safe and dummy usable user agent and xms user agent

UserAgent_Pool

Use for randomly get header

UserAgenter

Generated User Agent and Scrap Proxy [http, socka4, socka5](Auto update)

agentic-vision

Binary web cartography for AI agents

agentql

Tiny Fish AgentQL Python Client

agentscraper

Agent-based web scraping with LLM integration

agentsearchcli

Give any AI agent the ability to search, crawl, and extract the web.

ai-gemini-scraper

A simple package for scraping websites and generating responses using Google's Generative AI.

aio-lanraragi

An asynchronous Python API client for LANraragi, written with aiohttp and pydantic for type validation.

allmusic

AllMusic review scraper

alt-fake-useragent

A Fixed and Updated Alternative for fake-useragent

anti-useragent

fake pc or app browser useragent, anti useragent, and other awesome tools

apify-haystack

Apify integration for Haystack: load Actor results as Documents, search the web, and give Agents access to Apify Actors.

apiverve-useragentgenerator

User Agent Generator creates realistic user agent strings for desktop, mobile, and tablet devices. Each request outputs the complete string with parsed browser and operating system names and versions, while paid plans add engine and CPU details.

apiverve-useragentparser

User Agent Parser parses user agent strings to extract browser, operating system, and hardware details. It identifies device types, CPU architecture, and bot traffic, so you can run user agent detection on every incoming HTTP request.

autogen-ext-snaprender

AutoGen tools for SnapRender Screenshot API: capture websites as PNG, JPEG, WebP, or PDF

auxn-agent

A web scraping and data extraction tool.

azureaicommunity-agent-youtube-search

YouTube video search tools for AI agent pipelines

browser-useragent

get a browser user_agent by random

browserbase-haystack

Browserbase Haystack Fetcher

call-useragent

由于国内网络限制无法使用 fake_useragent 库而制作的替代品

clean-html-for-llm

A library for cleaning HTML content by removing specified tags and attributes.

cnki-agent

A Python package for scraping CNKI papers.

cortex-agent

Thin client for the Cortex web cartography runtime — map, compile, and query websites from Python

crawl-agent
crawler-user-agents

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders.

crewai-browseaidev

DEPRECATED - renamed to crewai-lastsearch. https://lastsearch.ai/migrate

crewai-browserless

A CrewAI tool for smart web scraping via the Browserless /smart-scrape API.

crewai-crw

CRW web scraping tools for CrewAI — scrape, crawl, and map websites

crewai-olostep

CrewAI tools for web scraping and research using Olostep API

crewai-tzafon

CrewAI tool for loading web pages using Tzafon's headless browser infrastructure

ddragon

Host your own ddragon mirror for League of Legends static data and images, relying on dragontail

device-user-agent

Parses user agent string built with the device_user-agent flutter package.

devtools-user-agent

User agent parser

django-smartagent

django-smartagent is the fastest and most complete user agent parser

django-user-agents

A django package that allows easy identification of visitors' browser, operating system and device information (mobile phone, tablet or has touch capabilities).

django-user-agents-last

A django package that allows easy identification of visitors' browser, operating system and device information (mobile phone, tablet or has touch capabilities).

django-user_agent_detector

Detect and handle desktop vs mobile user agents with django-annoying

django-useragents

A django package that allows easy identification of visitors' browser, operating system and device information (mobile phone, tablet or has touch capabilities).

dragnet

Extract the main article content (and optionally comments) from a web page

dragonball-germany

A python libary for dragonball-germany.de

dragonxxdlib

Download from tik tok and youtube and facebook and other soon

dummy-useragent

random useragent without network

fake-agent

Generate fake user agents on the fly

fake-random-useragent

A professional random User-Agent generator for web scraping.

fake-useragent

Up-to-date simple useragent faker with real world database

fake-useragent-ex

Up to date simple useragent faker with real world database

fake-useragent-kz

fake_useragent.

fake-useragent-migs

Up to date simple useragent faker with real world database

fake-useragxxx

Fake User-Agent Generator with Gray Randomization

fake-usrag-bor

Simple package with fake user agents for your requests.

fake_user_agent

Randomly generate a valid useragent for faking a browser.

firecrawl-haystack

Haystack integration for Firecrawl

forager

A utility for finding feeds on websites

foragerpy

The official Python client for Forager.

forge-useragent

Up-to-date simple user-agent faker with real world database

frag2text

Select and reverse-Markdown (html2text) web page fragments.

fragmentsoup

A wrapper for BeautifulSoup4 that restores the ability to work with HTML fragments

fragrantica-api
fresh-useragent

Simple User Agent Generator

garages-burgos

Asynchronous Python client for getting garage occupancy in Burgos

gc-user-agents

A library to identify devices (phones, tablets) and their capabilities by parsing (browser/HTTP) user agent strings

gemini-pro-web-scraper

This package contains implementation of a web scraper with Gemini Pro integrated into it.

generate-random-useragent

Random Useragent

geonode-scraper-crewai

Give CrewAI agents native Geonode web scraping tools to extract pages, search the web, discover URLs and manage crawl or batch jobs.

geonode-scraper-langchain

Give LangChain agents native Geonode web scraping tools to extract pages, search the web, discover URLs and manage crawl or batch jobs.

get-user-agent-pls

A package to fetch get-user-agent strings

get-useragents

This library is used to make it easier for developers or users to get random user agents

getuseragent

Get a random, popular, commonly-used user agent string. Random/fake/spoof user agents.

gpt-explorer

A package for gather the online data using GPT.

gpt-scraper

GPT-Scraper is an autonomous, LLM-based agent that generates code to extract structured information from web pages.

gpt-web-crawler

A web crawler for GPTs to build knowledge bases

gutefrage

Unofficial GuteFrage api. Check https://github.com/DAMcraft/gutefrage/wiki on how to use it

haystack-brightdata

Bright Data integration for Haystack - web scraping, SERP API, and data extraction from 45+ websites

hqdragondownload

CLI para baixar hqs no site.

httpagentparser

Extracts OS Browser etc information from http user agent string

hubstorage

Client interface for Scrapinghub HubStorage

More in Data, search & knowledge