Data, search & knowledge

Web scraping & crawling

676 agents · most starred first

Web scraping & crawling, page 5 of 7

hubstorage-frontera

Backend components for Frontera to use with Scrapinghub Hubstorage

ibrut-user-agent

Random instagram useragent app

karon-langchain

Karon Web Unblocker — Reserved package. See https://karonlabs.net/api/

langchain-advertools

LangChain integration for advertools

langchain-apify

An integration package connecting Apify and LangChain

langchain-brightdata

LangChain integration for Bright Data web scraping and SERP APIs

langchain-crawlbase

LangChain document loader, tool, and retriever backed by the Crawlbase API.

langchain-crawleo

An integration package connecting Crawleo (privacy-first search & crawler) and LangChain

langchain-crw

LangChain integration for CRW — high-performance web scraping document loader

langchain-firescraper

LangChain document loader for FireScraper — web scraping for AI pipelines.

langchain-mrscraper

LangChain tools for the MrScraper web-scraping API

langchain-nimble

An integration package connecting Nimble and LangChain

langchain-olostep

Live web search, scraping and crawling API for AI. Get structured, LLM-ready data from any site. Scale to 1M requests with 10 parallel batches at low cost.

langchain-oxylabs

Oxylabs integration package for LangChain

langchain-proxyclaw

LangChain integration for ProxyClaw - residential proxy network

langchain-pull-md

langchain-pull-md is a Python package that extends LangChain with a loader to convert URLs into Markdown. It addresses the challenge of extracting content from JavaScript-rendered pages, like those built with React, Angular, or Vue.js, by utilizing the pull.md service. This approach optimizes resource usage and ensures efficient, reliable Markdown conversion directly from URLs.

langchain-scavio

LangChain integration for the Scavio API -- 197 tools for real-time Google, YouTube, Amazon, Walmart, HEB, Kroger, Target, eBay, Home Depot, Reddit, TikTok, Instagram, X, Threads, Kuaishou, LinkedIn, Indeed, Glassdoor, Zillow, Redfin, Booking, Airbnb, Tripadvisor, Yelp, App Store, Google Play, SEC EDGAR, Companies House, G2, Capterra, Google Ads and Meta Ad Library data, plus extract, website map, crawl and batch extract tools

langchain-scrapegraph

Library for extracting structured data from websites using ScrapeGraphAI

langchain-scrapeless

An integration package connecting Scrapeless and LangChain

langchain-scraperapi

An integration package connecting ScraperAPI and LangChain

langchain-scrapingbee

An integration package connecting Scrapingbee and LangChain

langchain-scrapingpros

LangChain integration for Scraping Pros — web scraping tools for AI agents.

langchain-snaprender

LangChain tools for SnapRender Screenshot API: capture websites as PNG, JPEG, WebP, or PDF

langchain-trawl

LangChain integration for the Trawl unified content API — transcripts from 10+ sources

langchain-ujeebu

LangChain integration for Ujeebu: clean article content and web pages for LLM agents and RAG pipelines

langchain-wtmapi

LangChain Document Loader for WTM API — convert any webpage to Markdown

langchain-zenrows

A LangChain integration tool that provides reliable web scraping and AI-powered structured extraction at any scale using Zenrows' Fetch and Extract APIs

lanraragi_api

A Python library for LANraragi API.

latest-user-agents

Get the latest user agent strings for major browsers and OSs

lich_scrapy_random_useragent

Auto Generated by os-scrapy-cookiecutter

llama-index-packs-amazon-product-extraction

llama-index packs amazon_product_extraction integration

llama-index-readers-apify

llama-index readers apify integration

llama-index-readers-oxylabs

llama-index readers oxylabs integration

llama-index-readers-plasmate

LlamaIndex reader for Plasmate SOM, providing structured web content for AI agents

llama-index-readers-reddit

llama-index readers reddit integration

llama-index-readers-remote

llama-index readers remote integration

llama-index-readers-remote-depth

llama-index readers remote_depth integration

llama-index-readers-scrappey

llama-index readers scrappey integration

llama-index-readers-weather

llama-index readers weather integration

llama-index-readers-web

llama-index readers web integration

llama-index-readers-zyte-serp

llama-index readers zyte-serp integration

llama-index-tools-agentql

llama-index tools agentql integration

llama-index-tools-brightdata

llama-index tools Bright Data integration

llama-index-tools-jina

Project by kotko

llama-index-tools-quercle

Quercle tools for LlamaIndex - AI-powered web fetch and search

llama-index-tools-scrapegraphai

llama-index tools integrating ScrapegraphAI

llama-index-tools-scraperapi

llama-index tools to use ScraperAPI web scraping

llama-web-scraper

Web scraping tool potentially using Llama models.

llm-crawler

A web crawler CLI tool for preparing content for LLM ingestion and vector databases

llm-fragments-pure

Run URLs through the pure.md API

llm-fragments-reader

Run URLs through the Jina Reader API

llm-fragments-reddit

A plugin for llm to load Reddit threads as fragments

llm-fragments-site-text

Run URLs through the Trafilatura to get plain text

llm-hacker-news

LLM plugin for pulling content from Hacker News

llm-jina-reader

LLM plugin for pulling content from Jina Reader API

llm-scraper-py

Turn any webpage into structured data using LLMs

llm-url-markdown

Plugin for LLM that adds support for fetching markdown content from URLs using Jina Reader API

llm-webextract

AI-powered web content extraction with Large Language Models

llmify-cli

LLM-ify: create LLM-ready markdown and text from websites.

llmparser

Extract structured, LLM-ready content from any website — adaptive engine, no LLMs required

llms-sitemap-generator

AI-friendly llms.txt & sitemap generator for websites. Crawl or import sitemaps, filter pages, and generate curated indexes for LLMs.

llms-txt

The /llms.txt file, helping language models use your website

llmscraper

LLM-powered web scraping in Python

local-fake-useragent

Random request headers for local versions of browsers

local-useragent

fake_useragent use local data

meshagent-commoncrawl

Common Crawl import support for Meshagent datasets

meshagent-scrapy

Scrapy spider imports for Meshagent datasets

my-fake-useragent

create a fake useragent

newspaper3k-haystack

A simple newspaper3k haystack node wrapper. A node for scraping articles given a link and a crawler.

nxp-dlagent

Download agent for NXP material

opengrapher-useragent

Utility for parsing the OpenGraph tags - with customizable User-Agent headers

os-scrapy-random-useragent

Auto Generated by os-scrapy-cookiecutter

osm-integration-haystack

A Haystack component integrating Overpass API for OpenStreetMap

piki-rag-scrapper

Scrapper

pinballmap

Python client for the Pinball Map API

pragy-dl

A youtube-dl fork with additional features and patches

py-fake-useragent

Генерация User-Agent и HTTP-заголовков на ядре Go

py-random-useragent

Generate a random User Agent

python-tvrage

python client for the tvrage.com XML API

pyuser-agent

The User-Agent request header is a characteristic string that lets servers and network peers identify the application, operating system, vendor, and/or version of the requesting user agent.

pyuseragents

A Python module which does one thing: giving you a random User-Agent Header

r-useragent

generate random user-agent

ragalahari-dl

A powerful CLI tool to browse and bulk download HD photo galleries from Ragalahari.com

ragcrawl

Recursive website crawler producing LLM-ready knowledge base artifacts with Crawl4AI

raggy

scraping stuff

ragorez-scrapper

Scrapper

ragstoriches

Develop highly-concurrent web scrapers, easily.

rand-useragent

An up-to-date simple random user-agent with real world database.

random-user-agent

A package to get random user agents based filters provided by user

random-weighted-useragent

Randomly returns a weighted user agent for firewall evasion

randomagent

gets random useragent

randomuagents

Library for fetching random user agents based on device type (desktop, mobile, etc.).

real-useragent

Get Real user agent from auto update

requests-random-user-agent

Automatically generate a random User Agent for the requests library

scrapgpt

Python library for scraping ChatGPT

scrapy-agentfive-middleware

Scrapy downloader middleware to interact with agentfive API

scrapy-fake-useragent

Use a random User-Agent provided by fake-useragent for every request

scrapy-fake-useragent-fix

Use a random User-Agent provided by fake-useragent for every request

scrapy-feed-storage-internetarchive

Scrapy Item Feed Storage Backend for archive.org

scrapy-html-storage

Scrapy downloader middleware that stores response HTML files to disk.

More in Data, search & knowledge