Scrap agents

🔥 The Web Data API for AI - Power AI agents with clean web data

Create agents that monitor and act on your behalf. Your agents are standing by!

AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial…

Python scraper based on AI

An AI-powered research assistant that performs iterative, deep research on any topic by combining search…

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili…

Clone any website with one command using AI coding agents

Turn any webpage into structured data using LLMs

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM…

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as…

A community-driven way to read and chat with AI bots - powered by chatGPT.

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural…

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP…

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation…

Web crawler and scraper for Rust

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

(Supports DeepSeek R1) An AI-powered research assistant that performs iterative, deep research on any topic…

🔥 Visual workflow builder for AI agents powered by Firecrawl - drag-and-drop web scraping pipelines with…

Ui.Vision Open-Source RPA Software with Computer Vision, OCR, Anthropic Computer Use/LLM. Selenium IDE …

Get clean data from tricky documents, powered by vision-language models ⚡

Open-source MCP server for LinkedIn. Give Claude and any MCP-compatible AI assistant access to profiles…

Linkedin Automation Tool: Describe your product. Define your target market. The AI finds the leads for you.

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request…

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright…

🔥 AI-powered web monitoring platform. Create automated scouts that search the web and send email alerts when…

Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level…

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps…

🔥 This repository contains complete application examples, including websites and other projects, developed…

小红书版Openclaw，自媒体创作者的AI工作台，小红书创作AI工具RedClaw，支持小红书图文下载、创作风格学习、智囊团AI群聊、小红书AI创作，小红书内容打包下载等创作全程AI化，AI图文制作，AI文章排版，AI…

Turn any web app into an API. Chrome extension captures browser traffic, auto-generates schemas, lets AI…

Claude engineer that captures traffic, writes documentation and automatically generates API clients. Reverse…

AI Scraper is a powerful scraping tool and scrape agent built to automate data extraction with unmatched…

Free open-source Multilogin/Incogniton/Kameleo alternative for fingerprint spoofing…

Open-source, production-grade web scraping engine built for LLMs. Scrape and crawl the entire web, clean…

Official implement of paper "AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation"…

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust…

The only browser automation that bypasses anti-bot systems. AI writes network hooks, clones UIs pixel-perfect…

List of major web + mobile browser user agent strings. +1 Bonus script to scrape :)

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page…

Open-source AI agent for web automation and scraping.

Resume_Builder_AIHawk is a powerful Python tool that allows you to automatically customize your resume based…

ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.

Model Context Protocol (MCP) Server for Graphlit Platform

OpenClaw-inspired autonomous AI agent built entirely in n8n. Adaptive RAG-powered memory, Skills via MCP…

A completely private, locally-operated Ai Assistant/Chatbot/Sub-Agent Framework with realistic Long Term…

This AI Smart Speaker uses speech recognition, TTS (text-to-speech), and STT (speech-to-text) to enable voice…

Use LLMs to robustly extract web data

Tools to build web AI agents that can authenticate, interact with and extract data from any website.

📚 This is an adapted version of Jina AI's Reader for local deployment using Docker. Convert any URL to an…

AI agent that can SEE 👁️, control, navigate, & do stuff for you on your browser.

Extract knowledge from all information sources using gpt and other language models. Index and make Q&A…

Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web…

CLI and Agent Skill for Firecrawl - Add scrape, search, and browsing capabilities to your AI agents

The open-source execution engine for AI agents. 412 modules, MCP-native, triggers, queue, versioning…

Full-content web fetcher for AI agents — Chrome TLS fingerprinting, browser impersonation, and …

High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using…

Give your AI the power to browse, scrape, and extract structured data from complex websites — with faster…

Lego AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.

JRVS AI Agent with JARCORE autonomous coding engine - RAG knowledge base, web scraping, calendar, code…

This project provides a powerful web scraping tool that fetches search results and converts them into…

PocketGroq is a powerful Python library that simplifies integration with the Groq API, offering advanced…

Developed an AI application using LLM to analyze user resumes and provided the summarization, strengths…

⚡ The Complete X/Twitter Automation Toolkit — Scrapers, MCP server for AI agents (Claude/GPT), CLI, browser…

Unofficial Claude API supporting direct HTTP chat creation/deletion/retrieval, messages with multiple file…

An implementation of Google Deep Search 🕵️ with support for 1000+ references, local inference, chatting with…

Convert Files / Folders / GitHub Repos Into AI / LLM-ready Files

Model Context Protocol server that integrates AgentQL's data extraction capabilities.

AI-powered tool to audit and optimize website content by crawling URLs, analyzing H1s, and generating…

Renders any URL via headless Chrome, tiles screenshots into OCR slices, and streams structured Markdown +…

Scrape data from social media and chat with it using Langchain

Python package for webscraping in Natural language

skilless.ai gives your AI Agents real data capabilities - web search, web scraping, video download/subtitle…

Turn topics, links, and files into AI-generated research notebooks — summarize, explore, and ask anything.

OpenClaw skill for scraping any URL using the Decodo Web Scraping API.

AI tool for automating Upwork job applications using AI agents to find and qualify jobs, write personalized…

A high level scripting API for bot builders, developers, and maintainers.

Scrappy assistant that automates web3 bug hunting workflows. Tracks ongoing bug bounties and launches…

Intelligent Python system that extracts real estate property data as structured JSON using AI agents, Nebius…

GitHub Project: AI Job Application Automation 🚀 This project automates job searching, CV creation, and…

A powerful MCP server extension providing web search and content extraction capabilities. Integrates…

Token-efficient browser MCP server — structured web pages for AI agents, not raw accessibility dumps

A useful drawer for MacOS. chatting, clipboard, webscraping, window managing, shotcuts. built with Rust and …

CLI tool for agents to quickly access browser telemetry (DOM, network, console) via Chrome DevTools Protocol.

Automated Miulti AI Agent for company research with LangGraph— scrapes web data, extracts business insights…

Parse XML sitemaps and extract URLs. Designed to process millions of URLs while bypassing most modern…

Open‑source alternative to Perplexity Comet, director.ai and firecrawl combined

wxpath - declarative web crawling with XPath; a Web Query Language (WQL)

Battle-tested skill library for AI agents. Save 98% of API costs with ready-to-use code for crypto, PDFs…

Streamlit demo of Scrapegraph-ai for GPT4-hackaton

AURA (Agent-Usable Resource Assertion) is an open protocol designed to make the web machine-readable. It…

AI Browser Automation Framework

Erotic conversations scraped from public resources on the internet

Official Oxylabs MCP integration

Civic Tech & Data AI For Good project. Tracks prosecutor election messaging, mass incarceration indicators…

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation…

ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on…

AI-powered browser automation for Go — describe tasks in plain English, let the agent handle the clicks.

linkedin-jobs-RAG

一句话监控网页内容变化，AI | 爬虫 | 网页监控 | 网页更新提醒 | 网页内容订阅

CLI, MCP server, and npm library that turns any website into an API — no docs, no SDK, no browser.

Configures the requests library to randomly select a desktop User-Agent

pebkac Chrome Nonautomation - A Local LLM-Driven Web Co-Browser using Smolagents, Zendriver, Trafilatura.

AI web scraper built with Crawl4AI for extracting structured leads data from websites.

🕷️ A lightweight Model Context Protocol (MCP) server that exposes Crawl4AI web scraping and crawling…

JARVIS: a real-time agentic intelligence-gathering platform powered by autonomous web scraping & OSINT…

GPT4-powered Slack bot that can scrape URL contents

Extract data from websites in LLM ready JSON or CSV format. Crawl or Scrape entire website with Website…

MCP server for scraping LinkedIn, Facebook, Instagram profiles and Google search.

Whiskey AI lets you create autonomous AI agents without code. Connect APIs, automate Solana actions, launch…

Conversational agent that fuses chat data with live web results through Tavily search, extract, and crawl.

RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines with up-to-date text…

Official Python SDK for the ScrapeGraph AI API. Smart scraping, search, crawling, markdownify, agentic…

Anti-detection browser server for AI agents — REST API wrapping Camoufox engine with OpenClaw plugin support

⚡ Build structured YouTube datasets at scale — effortlessly fetch transcripts and rich metadata for NLP, ML…

The Offline Internet.

A (really) easy way to web scrape

Reddit_Commentator_AIHawk is a Python project showcasing the power of artificial intelligence in social media…

Automated Deep Research with LLMs, web search, paper parsing, and didactic summarization.

A unified web extraction and stateful automation engine for AI. Replaces heavy testing frameworks with…

This repo provides guidance on setting up a bedrock agent to webscrape and internet search via action groups

A simple, easy to use framework for adding randomized, anonymous IP addresses and user-agents to web…

Basic setup with random user agents and IP addresses for Python Scrapy Framework.

A collection of cookbooks to help developers get started quickly with the Firecrawl API.

Supacrawler's ultralight engine for scraping and crawling the web. Written in go for maximum performance and…

Crawlbase MCP Server connects AI agents and LLMs with real-time web data. It powers Claude, Cursor, and…

Fast, lightweight Firecrawl alternative in Rust. Web scraper, crawler & search API with MCP server for AI…

Random User Agent Generator library for Python, JS, TS, Rust, Go. Generate realistic browser user agents for…

Structured data gathering from any website using AI-powered scraper, crawler, and browser automation…

AI Lead Generation Agent that automatically discovers and qualifies potential leads from Quora. Using…

Retrieval-augmented generation (RAG) for remote & local LLM use

X (Twitter) data platform skill for AI coding agents. 122 REST API endpoints, 2 MCP tools, 23 extraction…

AI-powered agent that scrapes leads with Bright Data, qualifies them using OpenAI, and delivers…

Chew is a Go library for processing various content types into markdown/plaintext.

Qurio brings multi-provider models, custom agents, reusable skills, MCP servers, HTTP tools, retrieval…

Anti-detection browser MCP server for AI agents — navigate, interact, and automate the web without getting…

Modern JAV metadata manager — multi-source scraping, Jellyfin integration, and AI-ready API. Built with…

Reddit AI Agent is an intelligent tool that helps you explore Reddit like never before! 🔎 It allows you to…

LLM orchestration toolkit for agent workflows: planner + workers + synthesis, optional router (LLM + learned …

Create your own Alibaba dataset and interact with it in plain English.

Convert Files / Folders / GitHub Repos Into AI / LLM-ready Files

A powerful integration that combines Browserbase's Stagehand with Mastra for advanced web automation…

A chatbot demo that scrapes a website and stores the result in a vector db, which can then be queried via…

AI web agent to find answers to any question

We have developed a fully AI/ML-based itinerary recommendation system which when used by people coming to…

Build web scraping agents using AI to auto-extract the data from websites, capture screenshot, generate pdf…

Collection of Project Tutorials / blogs in python, web applications, machine learning, data science, deep…

ChatGPT selenium scraper written in Python

The official Python library for the Steel API

CrawlLama 🦙 is an local AI agent that answers questions via Ollama and integrates web- and RAG-based…

🔍 Model Context Protocol (MCP) tool for parsing websites using the Jina.ai Reader

Scraping Wikipedia by combining LangChain's agents and tools with OpenAI's LLMs and function calling

A tool to scrape all files from a GitHub repository and turn it into a JSON or TXT file, Useful for AI and…

This project is a Python script that scrapes a Linkedin PDF, generates a customized portfolio site using…

Extract Google Maps business leads and enrich contact details using AI & web scraping

How to guides on web-crawling or scraping

This is a template repository for building a web scraper with OpenAI support. The repository provides a basic…

GPT-3.5-ON-STEROIDS combines GPT with Python tools, empowering dynamic web scraping, language processing, and…

Pyvigate: A Python framework that combines headless browsing with LLMs that assists you in your data…

MCP Server leveraging crawl4ai for web scraping and LLM-based content extraction (Markdown, text snippets…

Just mention want you want and it will extract/scrape data from the Web. Useful to create AI web…

[IEEE S&P'26] WebCloak: Characterizing and Mitigating the Threats of LLM-Driven Web Agents as Intelligent…

Zero-dependency browser automation CLI. 70+ commands, 10 test assertions, smart commands (click/fill by text…

AI-driven tool for automatically collecting, analyzing, and generating actionable insights from customer…

The AI-native browser automation library. Snapshot + ref targeting — born from OpenClaw, built for agents, by…

AgentStack is a production-grade multi-agent framework built on Mastra, delivering 50+ enterprise tools, 25+…

Real-time Google Search API for AI Agents & RAG pipelines. Get structured SERP data instantly using remote…

Build a Reddit Content Research Agent with LLMs, LangChain, SERP, Jupyter, Django, Bright Data, Celery…

A :robot: which provides features from Wikipedia like summary, title searches, location API etc.

Parse SaaS pricing page using Open AI - GPT-3.5

Python, Javascript, and Rust libraries for the Spider Cloud API.

Intelligent stealth browser MCP server for AI agents with 30 tools, 22 anti fingerprint scripts, and LLM…

scraping and querying documents for LLMs

A web automation library that lets AI agents browse the web using declarative JSON, with a unified…

The unified web layer for AI agents. Search (8 engines), stealth browse, auth, and act on 24 platforms. One…

AI-powered deep research tool leveraging web scraping for cost-effective, comprehensive analysis. Open-source…

Turn any website into clean, LLM-ready data. Open-source web crawler with stealth mode, distributed crawling…

OSINT Skill for AI agents (Claude Code, OpenClaw, Codex, OpenCode) — from a name to a scored dossier with…

⚡️ Real-time Knowledge Graph for AI Agents. Connect LLMs to verified weather, stock, and currency data via…

AgentQL's integrations with workflow automation tools and AI agent frameworks let you extract structured data…

A Python project that extracts data from websites with the option to process the data through @openai's…

The Pudim Hunter 🍮 is a Proof of Concept (PoC) tool to scrape job listings from SimplyHired, analyze them…

Give your AI Agent the power to control Safari on macOS. No extensions, no separate browser.

Google's Bard ChatBot Unofficial NodeJS API

⚙️ List of Crew-Ai 👨‍👨‍👦‍👦 Tools 🛠️ (Webscrapper, PDF-RAG, Composio, YouTube-RAG...)

This repository demonstrates how to leverage OpenAI's GPT-4 models with JSON Strict Mode to extract…

An AI Integrated Chrome extension that generates the best cover letter by linking your resume and the scraped…

Crawl any website and convert it to clean, AI-ready Markdown — async Python CLI with MCP support, crawl…

Delegate complex tasks to Manus AI - web research, report generation, code building, data scraping. Task…

Web scraping skill for Claude AI. Crawl websites, extract structured data with CSS/LLM strategies, handle…

The browser engine for agents. HTML in, Semantic Object Model out. 10x token compression, V8 JS rendering…

An Artificial Intelligence Chat Bot and Service Provider written in Python and AIML.

RAG system with real-time news scraping built using mixtral-8x7b, ChromaDB, bart summarizer

The official Node.js / Typescript library for the Steel API

My personal Telegram bot made in Python. It has several features and it's based on Pyrogram.

AI-powered web scraping agent built with LangGraph, LangSmith, Firecrawl, and Anthropic AI. Automates…

This is a darkweb forums tracker that monitors forum posts and sends alerts to Discord