Category
Data, search & knowledge
Agents and libraries that find, retrieve and analyse information. Retrieval-augmented generation is the largest group, followed by vector databases, knowledge graphs, text-to-SQL analysts, web scrapers and document parsers. These are the pieces that let a model answer from your own data instead of from memory.
11,544 agents · 8 subcategories · mostly Python, Typescript, Jupyter notebook
Data, search & knowledge by type
- ragflow · RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engi
- llm-app · Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with l
- serviette · A no-code RAG server that keeps answers in sync with your documents. Works with
Data pipelines & ETL
1,206- dolt · Dolt – Git for Data
- airbyte · Open-source data movement for ELT pipelines and AI agents — from APIs, databases
- OpenMetadata · The Open Context Layer for Data and AI , OpenMetadata is the open platform for
- milvus · Milvus is a high-performance, cloud-native vector database built for scalable ve
- tidb · TiDB is built for agentic workloads that grow unpredictably, with ACID guarantee
- chroma · Search infrastructure for AI
- markitdown · Python tool for converting files and office documents to Markdown.
- PaddleOCR · Turn any PDF or image document into structured data for your AI. A powerful, lig
- MinerU · Transforms complex documents like PDFs and Office docs into LLM-ready markdown/J
- minds-platform · Platform dedicated to building an open foundation for applied Artificial Intelli
- vanna · 🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs usin
- prophet · Tool for producing high quality forecasts for time series data that has multiple
- firecrawl · Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ da
- Agent-Reach · Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddi
- crawl4ai · Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for A
- graphify · AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, and
- graphrag · A modular graph-based Retrieval-Augmented Generation (RAG) system
- auto-ontology · Auto Ontology is a Python framework for building semantic, graph-based reasoning
- Vane · Vane is an AI-powered answering engine.
- vespa · The AI search platform
- LLocalSearch · LLocalSearch is a completely locally running search aggregator using LLM Agents.
Every agent in Data, search & knowledge, most starred first
Knowledge infrastructure to make your agents smarter. Search, scrape and 550+ data providers through one API.
Python tool for converting files and office documents to Markdown.
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
RAGFlow is an agent-native database: a Retrieval-Augmented Generation (RAG) engine that grew into context infrastructure.
Turn any PDF or image document into structured data for your AI. A powerful, lightweight OCR toolkit that bridges the gap between images/PDFs and LLMs. Supports 100+ languages.
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.
Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.
Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data. 🐳Docker-friendly.⚡Always in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.
A no-code RAG server that keeps answers in sync with your documents. Works with 8 vector databases. Built on Pathway.
Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.
AI coding assistant skill (Claude Code, Codex, OpenCode, Cursor, Gemini CLI, and more). Turn any folder of code, SQL schemas, R scripts, shell scripts, docs, papers, images, or videos into a queryable knowledge graph. App code + database schema + infrastructure in one graph.
LlamaIndex is the document processing platform for AI
Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search
[EMNLP2025] LightRAG: Simple and Fast Retrieval-Augmented Generation
TiDB is built for agentic workloads that grow unpredictably, with ACID guarantees and native support for transactions, analytics, and vector search. No data silos. No noisy neighbors. No infrastructure ceiling.
📑 PageIndex: Document Index for Vectorless, Reasoning-based RAG
An open-source engine that turns continuous content streams into search and monitoring. Durable ingestion, hybrid search, alerts, and plugins for formats, models and business rules.
Opiniated RAG for integrating GenAI in your apps 🧠 Focus on your product rather than the RAG. Easy integration in existing products with customisation! Any LLM: GPT4, Groq, Llama. Any Vectorstore: PGVector, Faiss. Any Files. Anyway you want.
Opiniated RAG for integrating GenAI in your apps 🧠 Focus on your product rather than the RAG. Easy integration in existing products with customisation! Any LLM: GPT4, Groq, Llama. Any Vectorstore: PGVector, Faiss. Any Files. Anyway you want.
Platform dedicated to building an open foundation for applied Artificial Intelligence, designed for people seeking production-ready AI systems they can truly control, extend and deploy anywhere.
Langchain-Chatchat(原Langchain-ChatGLM)基于 Langchain 与 ChatGLM, Qwen 与 Llama 等语言模型的 RAG 与 Agent 应用 | Langchain-Chatchat (formerly langchain-ChatGLM), local knowledge based LLM (like ChatGLM, Qwen and Llama) RAG and Agent app with langchain
Vane is an AI-powered answering engine.
Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.
A modular graph-based Retrieval-Augmented Generation (RAG) system
Open Source AI Platform - AI Chat with advanced features that works with every LLM
Python scraper based on AI
This repository showcases various advanced techniques for Retrieval-Augmented Generation (RAG) systems. Each technique has a detailed notebook tutorial.
PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.
FastGPT is a knowledge-based platform built on the LLMs, offers a comprehensive suite of out-of-the-box capabilities such as data processing, RAG retrieval, and visual AI workflow orchestration, letting you easily develop and deploy complex question-answering systems without the need for extensive setup or configuration.
Search infrastructure for AI
An open-source RAG-based tool for chatting with your documents.
Dolt – Git for Data
🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs using Agentic Retrieval 🔄.
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Chat with your documents on your local device using GPT models. No data leaves your device and 100% private.
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.
Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.
open-source agentic AI data assistant for the next generation of AI + Data products.
AI search engine and MCP server for bots and agents. One Rust binary indexes code, docs, logs, JSONL/CSV and PDFs for BM25 full-text, vector and hybrid search, code search without grep, agent memory and RAG. Elasticsearch-compatible API; Corpus Hub serves hundreds of reference memory corpora.
Private AI platform for agents, assistants and enterprise search. Built-in Agent Builder, Deep research, Document analysis, Multi-model support, and API connectivity for agents.
GenBI (Generative BI) for AI agents, an open-source, governed text-to-SQL through an open context layer that turns natural-language questions into trusted dashboards, charts, and SQL across 20+ data sources, such as BigQuery, Snowflake, PostgreSQL, ClickHouse, Amazon Redshift, Databricks and more.
A vector index built on TurboQuant, written in Rust with Python bindings
Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your documents. Join our Discord: https://discord.gg/ejRNvftDp9
AI PDF chatbot agent built with LangChain & LangGraph
A lightweight, lightning-fast, in-process vector database
Apache Doris is a real-time analytics and hybrid search database for AI agents.
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.
:file_folder: Universal File Management Platform (for humans and agents)
Unified framework for building enterprise RAG pipelines with small, specialized models
Database governance built for humans and agents — controlling changes and access across every major database.
Auto Ontology is a Python framework for building semantic, graph-based reasoning layers over data. It enables deterministic query generation, better context retrieval, and more explainable AI for LLM and agent workflows.
[MLsys2026 Best Paper]: https://arxiv.org/abs/2506.08276. RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.
💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
🔍大模型应用开发实战一:RAG 技术全栈指南,在线阅读地址:https://datawhalechina.github.io/all-in-rag/
🏄 Scalable embedding, reasoning, ranking for images and sentences with CLIP
Incremental engine for long horizon agents 🌟 Star if you like it!
[MLsys2026]: RAG on Everything with LEANN. Enjoy 97% storage savings while running a fast, accurate, and 100% private RAG application on your personal device.
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/
A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 60+ detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.
High accuracy RAG for answering questions from scientific documents with citations
The fast, precise document-intelligence engine for every language: text, tables and metadata from 100+ formats, with OCR, transcription and code intelligence.
Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Deeplake is AI Data Runtime for Agents. It provides serverless postgres with a multimodal datalake, enabling scalable retrieval and training.
World's first open-source enterprise world model.
A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 97+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.
SoTA production-ready AI retrieval system. Agentic Retrieval-Augmented Generation (RAG) with a RESTful API.
A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.
可私有部署的多租户知识智能体平台:统一 RAG、知识图谱、多智能体、MCP/Skills、沙盒与权限管理。Yuxi = Cloud Agents + Knowledge RAG, Self-hosted knowledge agent platform for RAG, knowledge graphs and multi-agent workflows.
Local-first chat history analyzer with AI. | 本地优先的 AI 聊天记录分析工具
LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
The Open Source Feature Store for AI/ML
The AI search platform
🔥 基于大模型和 RAG 的智能问数系统,对话式数据分析神器。Text-to-SQL Generation via LLMs using RAG.
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
Turn any webpage into structured data using LLMs
Data visualization Skill for AI Agents, turning data into polished, interactive HTML charts. 面向 AI Agents 的数据可视化 Skill,将数据快速生成精致、可交互的 HTML 图表。
Postgres with GPUs for ML/AI apps.
Build ChatGPT over your data, all with natural language
Open-source context retrieval layer for AI agents
The SQL IDE for Your Terminal. Now with hsql, your agent's favorite sql client.
The live data layer for apps and AI agents. Create up-to-the-second views into your business, just using SQL
HelixDB is an OLTP graph database with native vector and full-text search built in Rust on Object Storage.
Agentic BI. Analytics at the speed of code ⚡️
The first "code-first" agent framework for seamlessly planning and executing data analytics tasks.
Your Personal Librarian Agent. Ask anything on your computer, organize files, remember important info and files.
LLocalSearch is a completely locally running search aggregator using LLM Agents. The user can ask a question and the system will use a chain of LLMs to find the answer. The user can see the progress of the agents and the final answer. No OpenAI or Google API keys are needed.
Chat-based SQL Client and Editor for the next decade
A suite of tools to develop RAG, semantic search, and other AI applications more easily with PostgreSQL
A Low-Code MCP Framework for Building Complex and Innovative RAG Pipelines
An AI-powered data science team of agents to help you perform common data science tasks 10X faster.
Neo4j graph construction from unstructured data using LLMs
This tool has been deprecated. Use Agentic Document Extraction instead.
Structured data extraction, instruction calling and agentic workflows with ML, LLM and Vision LLM
Where data access meets operational intelligence
Perplexity Inspired Answer Engine
Questions
What is RAG?
Retrieval-augmented generation: before the model answers, the system searches a store of your documents and puts the most relevant passages into the prompt. It keeps answers grounded and current without retraining the model.
Do I need a vector database for RAG?
Not always. Small collections work with a library-level index or even plain keyword search. A dedicated vector database pays off when you have many documents, need filtering or hybrid search, or serve many users at once.