Web scraping & crawling
676 agents · most starred first
Web scraping & crawling, page 5 of 7
Backend components for Frontera to use with Scrapinghub Hubstorage
Random instagram useragent app
Karon Web Unblocker — Reserved package. See https://karonlabs.net/api/
LangChain integration for advertools
An integration package connecting Apify and LangChain
LangChain integration for Bright Data web scraping and SERP APIs
LangChain document loader, tool, and retriever backed by the Crawlbase API.
An integration package connecting Crawleo (privacy-first search & crawler) and LangChain
LangChain integration for CRW — high-performance web scraping document loader
LangChain document loader for FireScraper — web scraping for AI pipelines.
LangChain tools for the MrScraper web-scraping API
An integration package connecting Nimble and LangChain
Live web search, scraping and crawling API for AI. Get structured, LLM-ready data from any site. Scale to 1M requests with 10 parallel batches at low cost.
Oxylabs integration package for LangChain
LangChain integration for ProxyClaw - residential proxy network
langchain-pull-md is a Python package that extends LangChain with a loader to convert URLs into Markdown. It addresses the challenge of extracting content from JavaScript-rendered pages, like those built with React, Angular, or Vue.js, by utilizing the pull.md service. This approach optimizes resource usage and ensures efficient, reliable Markdown conversion directly from URLs.
LangChain integration for the Scavio API -- 197 tools for real-time Google, YouTube, Amazon, Walmart, HEB, Kroger, Target, eBay, Home Depot, Reddit, TikTok, Instagram, X, Threads, Kuaishou, LinkedIn, Indeed, Glassdoor, Zillow, Redfin, Booking, Airbnb, Tripadvisor, Yelp, App Store, Google Play, SEC EDGAR, Companies House, G2, Capterra, Google Ads and Meta Ad Library data, plus extract, website map, crawl and batch extract tools
Library for extracting structured data from websites using ScrapeGraphAI
An integration package connecting Scrapeless and LangChain
An integration package connecting ScraperAPI and LangChain
An integration package connecting Scrapingbee and LangChain
LangChain integration for Scraping Pros — web scraping tools for AI agents.
LangChain tools for SnapRender Screenshot API: capture websites as PNG, JPEG, WebP, or PDF
LangChain integration for the Trawl unified content API — transcripts from 10+ sources
LangChain integration for Ujeebu: clean article content and web pages for LLM agents and RAG pipelines
LangChain Document Loader for WTM API — convert any webpage to Markdown
A LangChain integration tool that provides reliable web scraping and AI-powered structured extraction at any scale using Zenrows' Fetch and Extract APIs
A Python library for LANraragi API.
Get the latest user agent strings for major browsers and OSs
Auto Generated by os-scrapy-cookiecutter
llama-index packs amazon_product_extraction integration
llama-index readers apify integration
llama-index readers oxylabs integration
LlamaIndex reader for Plasmate SOM, providing structured web content for AI agents
llama-index readers reddit integration
llama-index readers remote integration
llama-index readers remote_depth integration
llama-index readers scrappey integration
llama-index readers weather integration
llama-index readers web integration
llama-index readers zyte-serp integration
llama-index tools agentql integration
llama-index tools Bright Data integration
Project by kotko
Quercle tools for LlamaIndex - AI-powered web fetch and search
llama-index tools integrating ScrapegraphAI
llama-index tools to use ScraperAPI web scraping
Web scraping tool potentially using Llama models.
A web crawler CLI tool for preparing content for LLM ingestion and vector databases
Run URLs through the pure.md API
Run URLs through the Jina Reader API
A plugin for llm to load Reddit threads as fragments
Run URLs through the Trafilatura to get plain text
LLM plugin for pulling content from Hacker News
LLM plugin for pulling content from Jina Reader API
Turn any webpage into structured data using LLMs
Plugin for LLM that adds support for fetching markdown content from URLs using Jina Reader API
AI-powered web content extraction with Large Language Models
LLM-ify: create LLM-ready markdown and text from websites.
Extract structured, LLM-ready content from any website — adaptive engine, no LLMs required
AI-friendly llms.txt & sitemap generator for websites. Crawl or import sitemaps, filter pages, and generate curated indexes for LLMs.
The /llms.txt file, helping language models use your website
LLM-powered web scraping in Python
Random request headers for local versions of browsers
fake_useragent use local data
Common Crawl import support for Meshagent datasets
Scrapy spider imports for Meshagent datasets
create a fake useragent
A simple newspaper3k haystack node wrapper. A node for scraping articles given a link and a crawler.
Download agent for NXP material
Utility for parsing the OpenGraph tags - with customizable User-Agent headers
Auto Generated by os-scrapy-cookiecutter
A Haystack component integrating Overpass API for OpenStreetMap
Scrapper
Python client for the Pinball Map API
A youtube-dl fork with additional features and patches
Генерация User-Agent и HTTP-заголовков на ядре Go
Generate a random User Agent
python client for the tvrage.com XML API
The User-Agent request header is a characteristic string that lets servers and network peers identify the application, operating system, vendor, and/or version of the requesting user agent.
A Python module which does one thing: giving you a random User-Agent Header
generate random user-agent
A powerful CLI tool to browse and bulk download HD photo galleries from Ragalahari.com
Recursive website crawler producing LLM-ready knowledge base artifacts with Crawl4AI
scraping stuff
Scrapper
Develop highly-concurrent web scrapers, easily.
An up-to-date simple random user-agent with real world database.
A package to get random user agents based filters provided by user
Randomly returns a weighted user agent for firewall evasion
gets random useragent
Library for fetching random user agents based on device type (desktop, mobile, etc.).
Get Real user agent from auto update
Automatically generate a random User Agent for the requests library
Python library for scraping ChatGPT
Scrapy downloader middleware to interact with agentfive API
Use a random User-Agent provided by fake-useragent for every request
Use a random User-Agent provided by fake-useragent for every request
Scrapy Item Feed Storage Backend for archive.org
Scrapy downloader middleware that stores response HTML files to disk.