Category

Data, search & knowledge

Agents and libraries that find, retrieve and analyse information. Retrieval-augmented generation is the largest group, followed by vector databases, knowledge graphs, text-to-SQL analysts, web scrapers and document parsers. These are the pieces that let a model answer from your own data instead of from memory.

11,544 agents · 8 subcategories · mostly Python, Typescript, Jupyter notebook

Data, search & knowledge by type

  • ragflow · RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engi
  • llm-app · Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with l
  • serviette · A no-code RAG server that keeps answers in sync with your documents. Works with
See all 5,258 →
  • dolt · Dolt – Git for Data
  • airbyte · Open-source data movement for ELT pipelines and AI agents — from APIs, databases
  • OpenMetadata · The Open Context Layer for Data and AI , OpenMetadata is the open platform for
See all 1,206 →
  • milvus · Milvus is a high-performance, cloud-native vector database built for scalable ve
  • tidb · TiDB is built for agentic workloads that grow unpredictably, with ACID guarantee
  • chroma · Search infrastructure for AI
See all 1,175 →
  • markitdown · Python tool for converting files and office documents to Markdown.
  • PaddleOCR · Turn any PDF or image document into structured data for your AI. A powerful, lig
  • MinerU · Transforms complex documents like PDFs and Office docs into LLM-ready markdown/J
See all 1,074 →
  • minds-platform · Platform dedicated to building an open foundation for applied Artificial Intelli
  • vanna · 🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs usin
  • prophet · Tool for producing high quality forecasts for time series data that has multiple
See all 1,053 →
  • firecrawl · Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ da
  • Agent-Reach · Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddi
  • crawl4ai · Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for A
See all 676 →
  • graphify · AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, and
  • graphrag · A modular graph-based Retrieval-Augmented Generation (RAG) system
  • auto-ontology · Auto Ontology is a Python framework for building semantic, graph-based reasoning
See all 669 →
  • Vane · Vane is an AI-powered answering engine.
  • vespa · The AI search platform
  • LLocalSearch · LLocalSearch is a completely locally running search aggregator using LLM Agents.
See all 411 →

Every agent in Data, search & knowledge, most starred first

firecrawl★ 204,752

Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.

markitdown★ 197,149

Python tool for converting files and office documents to Markdown.

Agent-Reach★ 113,459

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

ragflow★ 94,473

RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.

PaddleOCR★ 92,943

Turn any PDF or image document into structured data for your AI. A powerful, lightweight OCR toolkit that bridges the gap between images/PDFs and LLMs. Supports 100+ languages.

crawl4ai★ 87,593

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

MinerU★ 84,042

Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

llm-app★ 58,873

Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data. 🐳Docker-friendly.⚡Always in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.

serviette★ 58,699

A no-code RAG server that keeps answers in sync with your documents. Works with 8 vector databases. Built on Pathway.

private-gpt★ 57,645

Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.

graphify★ 54,492

AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, and more). Turn any folder of code, SQL schemas, R scripts, shell scripts, docs, papers, images, or videos into a queryable knowledge graph. App code + database schema + infrastructure in one graph.

llama_index★ 52,967

LlamaIndex is the document processing platform for AI

milvus★ 46,706

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

LightRAG★ 40,788

[EMNLP2025] LightRAG: Simple and Fast Retrieval-Augmented Generation

tidb★ 40,756

TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.

PageIndex★ 40,495

📑 PageIndex: Document Index for Vectorless, Reasoning-based RAG

quivr★ 39,655

An open-source engine that turns continuous content streams into search and monitoring. Durable ingestion, hybrid search, alerts, and plugins for formats, models and business rules.

Quivr★ 39,440

Opiniated RAG for integrating GenAI in your apps 🧠 Focus on your product rather than the RAG. Easy integration in existing products with customisation! Any LLM: GPT4, Groq, Llama. Any Vectorstore: PGVector, Faiss. Any Files. Anyway you want.

quivr★ 39,438

Opiniated RAG for integrating GenAI in your apps 🧠 Focus on your product rather than the RAG. Easy integration in existing products with customisation! Any LLM: GPT4, Groq, Llama. Any Vectorstore: PGVector, Faiss. Any Files. Anyway you want.

minds-platform★ 39,207

Platform dedicated to building an open foundation for applied Artificial Intelligence, designed for people seeking production-ready AI systems they can truly control, extend and deploy anywhere.

Langchain-Chatchat★ 38,715

Langchain-Chatchat(原Langchain-ChatGLM)基于 Langchain 与 ChatGLM, Qwen 与 Llama 等语言模型的 RAG 与 Agent 应用 | Langchain-Chatchat (formerly langchain-ChatGLM), local knowledge based LLM (like ChatGLM, Qwen and Llama) RAG and Agent app with langchain

Vane★ 38,267

Vane is an AI-powered answering engine.

WeKnora★ 37,874

Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.

graphrag★ 36,699

A modular graph-based Retrieval-Augmented Generation (RAG) system

onyx★ 33,482

Open Source AI Platform - AI Chat with advanced features that works with every LLM

Scrapegraph-ai★ 32,868

Python scraper based on AI

RAG_Techniques★ 30,444

This repository showcases various advanced techniques for Retrieval-Augmented Generation (RAG) systems. Each technique has a detailed notebook tutorial.

opendataloader-pdf★ 30,352

PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.

FastGPT★ 30,079

FastGPT is a knowledge-based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.

chroma★ 29,715

Search infrastructure for AI

kotaemon★ 25,925

An open-source RAG-based tool for chatting with your documents.

dolt★ 24,946

Dolt – Git for Data

vanna★ 23,790

🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs using Agentic Retrieval 🔄.

airbyte★ 22,422

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

localGPT★ 22,190

Chat with your documents on your local device using GPT models. No data leaves your device and 100% private.

Douyin_TikTok_Download_API★ 21,499

🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

prophet★ 20,460

Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.

DB-GPT★ 20,319

open-source agentic AI data assistant for the next generation of AI + Data products.

xerj★ 18,993

AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.

DocsGPT★ 18,375

Private AI platform for agents, assistants and enterprise search. Built-in Agent Builder, Deep research, Document analysis, Multi-model support, and API connectivity for agents.

WrenAI★ 18,351

GenBI (Generative BI) for AI agents, an open-source, governed text-to-SQL through an open context layer that turns natural-language questions into trusted dashboards, charts, and SQL across 20+ data sources, such as BigQuery, Snowflake, PostgreSQL, ClickHouse, Amazon Redshift, Databricks and more.

turbovec★ 17,503

A vector index built on TurboQuant, written in Rust with Python bindings

SurfSense★ 16,900

Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9

ai-pdf-chatbot-langchain★ 16,580

AI PDF chatbot agent built with LangChain & LangGraph

zvec★ 16,273

A lightweight, lightning-fast, in-process vector database

doris★ 16,243

Apache Doris is a real-time analytics and hybrid search database for AI agents.

OpenMetadata★ 15,823

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

unstructured★ 15,769

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

filestash★ 14,920

:file_folder: Universal File Management Platform (for humans and agents)

llmware★ 14,821

Unified framework for building enterprise RAG pipelines with small, specialized models

bytebase★ 14,723

Database governance built for humans and agents — controlling changes and access across every major database.

auto-ontology★ 14,017

Auto Ontology is a Python framework for building semantic, graph-based reasoning layers over data. It enables deterministic query generation, better context retrieval, and more explainable AI for LLM and agent workflows.

LEANN★ 13,806

[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.

txtai★ 13,667

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

all-in-rag★ 13,260

🔍大模型应用开发实战一:RAG 技术全栈指南,在线阅读地址:https://datawhalechina.github.io/all-in-rag/

clip-as-service★ 12,841

🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP

cocoindex★ 11,918

Incremental engine for long horizon agents 🌟 Star if you like it!

LEANN★ 11,761

[MLsys2026]: RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.

kedro★ 11,057

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

PixelRAG★ 10,969

https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/

production-agentic-rag-course★ 10,167
pyod★ 10,097

A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 60+ detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.

paper-qa★ 9,627

High accuracy RAG for answering questions from scientific documents with citations

xberg★ 9,602

The fast, precise document-intelligence engine for every language: text, tables and metadata from 100+ formats, with OCR, transcription and code intelligence.

databend★ 9,501

Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.

risingwave★ 9,425

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

deeplake★ 9,271

Deeplake is AI Data Runtime for Agents. It provides serverless postgres with a multimodal datalake, enabling scalable retrieval and training.

utopia★ 8,738

World's first open-source enterprise world model.

kreuzberg★ 8,397

A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

R2R★ 8,103

SoTA production-ready AI retrieval system. Agentic Retrieval-Augmented Generation (RAG) with a RESTful API.

azure-search-openai-demo★ 7,805

A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.

Yuxi★ 7,709

可私有部署的多租户知识智能体平台:统一 RAG、知识图谱、多智能体、MCP/Skills、沙盒与权限管理。Yuxi = Cloud Agents + Knowledge RAG, Self-hosted knowledge agent platform for RAG, knowledge graphs and multi-agent workflows.

ChatLab★ 7,677

Local-first chat history analyzer with AI. | 本地优先的 AI 聊天记录分析工具

unstract★ 7,461

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

feast★ 7,362

The Open Source Feature Store for AI/ML

vespa★ 7,186

The AI search platform

SQLBot★ 7,182

🔥 基于大模型和 RAG 的智能问数系统,对话式数据分析神器。Text-to-SQL Generation via LLMs using RAG.

trafilatura★ 7,155

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

llm-scraper★ 6,957

Turn any webpage into structured data using LLMs

lieflat-charts★ 6,941

Data visualization Skill for AI Agents, turning data into polished, interactive HTML charts. 面向 AI Agents 的数据可视化 Skill,将数据快速生成精致、可交互的 HTML 图表。

postgresml★ 6,831

Postgres with GPUs for ML/AI apps.

rags★ 6,547

Build ChatGPT over your data, all with natural language

airweave★ 6,541

Open-source context retrieval layer for AI agents

harlequin★ 6,530

The SQL IDE for Your Terminal. Now with hsql, your agent's favorite sql client.

chat-langchain★ 6,479
materialize★ 6,407

The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL

helix-db★ 6,304

HelixDB is an OLTP graph database with native vector and full-text search built in Rust on Object Storage.

lightdash★ 6,241

Agentic BI. Analytics at the speed of code ⚡️

TaskWeaver★ 6,154

The first "code-first" agent framework for seamlessly planning and executing data analytics tasks.

AutoRAG★ 6,081

Your Personal Librarian Agent. Ask anything on your computer, organize files, remember important info and files.

LLocalSearch★ 5,929

LLocalSearch is a completely locally running search aggregator using LLM Agents. The user can ask a question and the system will use a chain of LLMs to find the answer. The user can see the progress of the agents and the final answer. No OpenAI or Google API keys are needed.

sqlchat★ 5,854

Chat-based SQL Client and Editor for the next decade

pgai★ 5,790

A suite of tools to develop RAG, semantic search, and other AI applications more easily with PostgreSQL

UltraRAG★ 5,748

A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines

ai-data-science-team★ 5,513

An AI-powered data science team of agents to help you perform common data science tasks 10X faster.

llm-graph-builder★ 5,318

Neo4j graph construction from unstructured data using LLMs

vision-agent★ 5,298

This tool has been deprecated. Use Agentic Document Extraction instead.

sparrow★ 5,234

Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM

whodb★ 5,063

Where data access meets operational intelligence

llm-answer-engine★ 5,041

Perplexity Inspired Answer Engine

Questions

What is RAG?

Retrieval-augmented generation: before the model answers, the system searches a store of your documents and puts the most relevant passages into the prompt. It keeps answers grounded and current without retraining the model.

Do I need a vector database for RAG?

Not always. Small collections work with a library-level index or even plain keyword search. A dedicated vector database pays off when you have many documents, need filtering or hybrid search, or serve many users at once.

Frameworks used in Data, search & knowledge

Other categories