Data, search & knowledge

Data pipelines & ETL

Data pipelines and ETL projects move and reshape data: pulling records from APIs and databases, cleaning and transforming them, and keeping datasets versioned and traceable. Airbyte connects sources to warehouses and AI applications, while Dolt applies Git-style branching and history to tabular data.

1,206 agents · most starred first

Top 100 of 1,206

dolt★ 24,946

Dolt – Git for Data

airbyte★ 22,422

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

OpenMetadata★ 15,823

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

cocoindex★ 11,918

Incremental engine for long horizon agents 🌟 Star if you like it!

kedro★ 11,057

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

pyod★ 10,097

A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 60+ detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.

risingwave★ 9,425

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

feast★ 7,362

The Open Source Feature Store for AI/ML

docetl★ 4,264

A system for agentic LLM-powered data processing and ETL

datachain★ 2,831

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

hamilton★ 2,630

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

DataDesigner★ 2,418

🎨 NeMo Data Designer: Generate high-quality synthetic data from scratch or from seed data.

portaljs★ 2,372

🌀 AI-native framework for building data portals. Scaffold a full portal from a brief and load datasets in minutes with agentic skills — any backend (CKAN, GitHub, Frictionless).

autolabel★ 2,328

Label, clean and enrich text datasets with LLMs.

marquez★ 2,310

Collect, aggregate, and visualize a data ecosystem's metadata

proton★ 2,286

The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.

orbit★ 2,082

A Python package for Bayesian forecasting with object-oriented design and probabilistic models under the hood.

lotus★ 1,735

Optimized Agentic and LLM Bulk Processing Over Your Data

metacat★ 1,690
trench★ 1,673

Trench — Open-Source Analytics Infrastructure. A single production-ready Docker image built on ClickHouse, Kafka, and Node.js for tracking events. Easily build product analytics dashboards, LLM RAGs, observability platforms, or any other analytics product.

pixeltable★ 1,662

The backend agents build with - Multimodal database, orchestration, and serving in one file

duckle★ 1,577

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 437 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

lakekeeper★ 1,505

Apache Iceberg REST Catalog in Rust — access control, credential vending and audit for every engine and AI agent. Apache 2.0.

pycall.rb★ 1,119

Calling Python functions from the Ruby language

dataset-viewer★ 972

A sleek dataset viewer built entirely by AI Agent. Supports streaming large files from WebDAV, S3, SSH, Local or Hugging Face.

restheart★ 892

A backend for web, mobile, AI and IoT apps. Built on MongoDB

Chatito★ 890

🎯🗯 Dataset generation for AI chatbots, NLP tasks, named entity recognition or text classification models using a simple DSL!

vulcan-sql★ 792

Data API Framework for AI Agents and Data Apps

fenic★ 681

Semantic DataFrames for humans and agents

timecopilot★ 643

TimeCopilot: the Agentic Forecasting Framework. Built on LLMs and Time Series Foundation Models, it lets you forecast, cross-validate, and detect anomalies using multiple foundation models through a single API. From finance and energy to web analytics, TimeCopilot turns natural-language queries into production-ready forecasts.

sqlite-sync★ 464

CRDT-based offline-first sync for SQLite. Syncs automatically with SQLite Cloud, PostgreSQL, and Supabase. No conflicts, no data loss, no backend to build. For offline-first apps and AI agents.

agents★ 464

AI agent tooling for data engineering workflows.

potato★ 447

potato: the portable annotation tool

DAComp★ 436

[ICLR 2026] DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle

lamindb★ 436

Open-source data management for multimodal AI. Query, trace, and govern with a lineage-native, format-agnostic lakehouse for agents and teams. Supports biological formats and registries - by the creators of Scanpy. 🍊YC S22

datalinkx★ 379

🔥🔥DatalinkX异构数据源之间的数据同步系统,支持海量数据的增量或全量同步,同时支持HTTP、Oracle、MySQL、ES等数据源之间的数据流转,支持中间transform算子如SQL算子、大模型算子,底层依赖Flink、Seatunnel引擎,提供流转任务管理、任务级联配置、任务日志采集等功能🔥🔥

DataMate★ 375

DataMate is an enterprise-level data processing platform designed for model fine-tuning and RAG retrieval.

Jaiqu★ 346

Automatically reformat any JSON into any schema with AI

MuTual★ 334

A Dataset for Multi-Turn Dialogue Reasoning

prodigy-openai-recipes★ 319

✨ Bootstrap annotation with zero- & few-shot learning via OpenAI GPT-3

butterfree★ 319

A tool for building feature stores.

Chatette★ 316

A powerful dataset generator for Rasa NLU, inspired by Chatito

datasetGPT★ 300

A command-line interface to generate textual and conversational datasets with LLMs.

ezdata★ 294

AI-native data platform: 60+ connectors, dlt-based ETL, DAG scheduling, per-source RAG knowledge bases, and sandboxed AI agents that query, chart, and analyze your data.

isolation-forest★ 265

A distributed Spark/Scala implementation of the isolation forest and extended isolation forest algorithms for unsupervised outlier detection, featuring support for scalable training and ONNX export for easy cross-platform inference.

convoys★ 261

Implementation of statistical models to analyze time lagged conversions

data-enrichment★ 261

LangGraph Studio template for creating an agent that does web research to genearte or enrich structured data.

Gossiping-Chinese-Corpus★ 248

PTT 八卦版問答中文語料

question_extractor★ 237

Generate question/answer training pairs out of raw text.

sqrl★ 237

Agentic data engineering for operational data: real-time pipelines, data APIs, and data products on Kafka, Flink, Postgres, and Iceberg.

FuzzTypes★ 219

Pydantic extension for annotating autocorrecting fields.

agentic-kaggle-skill★ 193

🤖 AI Agent-driven Kaggle competition workflow. Battle-tested patterns for score stabilization, submission troubleshooting, kernel workflows, and spec-driven development.

agent-app-ingestion★ 187

Data ingestion service for the Agent Landing Zone. Processes PDFs, images, spreadsheets, transcripts and SharePoint content into grounding knowledge for agents, with smart chunking and multimodal embeddings.

syda★ 178

AI-powered synthetic data generation — structured tables, unstructured documents, multi-provider LLM support, referential integrity, and code-gen mode for millions of rows

sdg_hub★ 171

Synthetic Data Generation Toolkit for LLMs

gfs★ 164

Git For database Systems

avaloka★ 147

Open source release of Data Science Agents that can analyze and transform datasets as well as train and infer models to augment your data scientists.

beava★ 139

Real-time decision features without streaming infra. Turn live events into product reflexes — no Kafka, no Flink, no feature store.

data-engineering★ 126

🚀 Advanced Data & AI Engineering Portfolio: Real-world projects and production-ready patterns to level up your AI skills—from building clean data pipelines to deploying RAG systems, AI agents, and intelligent dashboards.

fountain★ 115

Natural Language Data Augmentation Tool for Conversational Systems

PyOmniTS★ 112

🔬 A Researcher&Agent-Friendly Framework for Time Series Analysis. Train Any Model on Any Dataset!

VectorETL★ 108

Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications

AD-AGENT★ 104

A multi-agent framework to fully automate anomaly detection in different modalities, tabular, graph, time series, and more (work in progress)!

mloda★ 101

mloda.ai - Open Data Access for AI and ML. Plugin-based. Traceable. Framework-agnostic.

AutoMind★ 100

AutoMind: Adaptive Knowledgeable Agent for Automated Data Science

DataTonic★ 95

🌟DataTonic : A Data-Capable AGI-style Agent Builder of Agents , that creates swarms , runs commands and securely processes and creates datasets, databases, visualisations, and analyses.

yolodex★ 95

agent skills for autonomous data labeling, winner at openai codex hackathon 2026

databricks-template★ 87

Production-ready PySpark ETL template for Databricks — medallion architecture, DABs, tests, DQX, CI/CD, and agentic development with Claude Code.

kaggle-Agent★ 86

🚀 Kaggle Problem Solver: AI-powered ML challenge automation Plan, code, and execute Kaggle solutions with intelligent agents. Boost your data science workflow! 🧠📊

meadow★ 83

Framework for building data agent workflows

interactive-map-builder★ 80

Build polished interactive maps and report-ready figures from spatial data with AI agents.

ProMoAI★ 79

An AI-powered process mining suite that converts natural language to process models and uses autonomous agents to analyze event logs and generate actionable process insights.

gutenberg-dialog★ 77

Build a dialog dataset from online books in many languages

flapi★ 75

API Framework heavily relying on the power of DuckDB and DuckDB extensions. Ready to build performant and cost-efficient APIs on top of BigQuery or Snowflake for AI Agents and Data Apps

visitran★ 70

Agentic & No-Code Data Transformations

trace-file-lineage★ 68

Find which script, notebook, data, command, or AI agent produced a file — locally, with evidence and honest uncertainty.

TreeExtra★ 67

Additive Groves, Bagged Trees with Feature Evaluation, Interaction Detection, Visualization of Feature Effects.

gcf★ 63

The AI-native wire format for structured data. 100% comprehension on every frontier model. 50-92% fewer tokens than JSON. 43B+ lossless round-trips across 17 formats. Spec v3.5.1 Stable.

data-juicer-agents★ 61

🤖 Your Intelligent Copilot for Data Exploration and Processing Pipeline

oracle-aidp-samples★ 61

Oracle AI Data Platform Workbench Samples

discus★ 60

A data-centric AI package for ML/AI. Get the best high-quality data for the best results. Discord: https://discord.gg/t6ADqBKrdZ

synthgenai★ 59

SynthGenAI - Package for Generating Synthetic Datasets using LLMs.

aws-data-science-data-engineering-mlops-infra★ 59

Ultimate AWS Data & AI Platform: Real-time flight delay predictions with complete DE, DS, MLOps, Web App & Multi-Agent LLM - All deployed via CDK self-mutating pipelines

doccano-mini★ 57

Annotation meets Large Language Models (ChatGPT, GPT-3 and alike).

dsh-data-quality★ 54

DeepSeek Harness plugin: deterministic data profiling, cleaning, and verification (dsh-data-quality)

role-play-synthetic★ 53

Synthetic Role-Play Conversation Dataset Generation

OmniTag-Forge★ 53

SmartLabelBench 2026: LLM-Powered Auto Annotation and Dataset Builder for Everything

DialogueAct-Tagger★ 52

A resource to create a multi domain Dialog Act Tagger for conversational agents using publicly available data

agentic_learning★ 50
fabric-task-flows★ 50

Task flows let you describe your problem and have an AI agent generate your Microsoft Fabric solution—workspaces, pipelines, notebooks, data agents, models, reports, and more—delivered as a CI/CD‑ready script, ready to deploy.

duckdb-differential★ 50

Versioned DuckDB databases, locally or replicated through S3, for people and agents

data-engineering-agent-skills★ 48

Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.

SciDataCopilot★ 47

An Agentic Data Preparation Framework for AGI-driven Scientific Discovery

isledb★ 47

A key-value database for object storage.

laravel-faker-openai★ 45

An opinionated Laravel package that extends FakerPHP and uses openai-php/laravel to generate fake data

menome_processor★ 44

worker agent API for processing data for Menome Knowledge Vault

Agentic-Medallion★ 44

Revolutionary AI ETL with Medallion Architecture: Zero-touch autonomous & HITL pipelines on Databricks

tabsdata★ 44

Agentic data engineering built on Pub/Sub for Tables. Discover, publish, transform and consume data as versioned tables with lineage, with an AI agent doing the assembly and the engineer approving.

Codify★ 43

Codify enables data scientists to perform all the tedious and time-consuming tasks such as EDA (exploratory data analysis), data cleaning, data pre-processing, data visualization, modeling, and evaluation in the data-science life cycle, by only conveying the logic of the task in natural language (English) and the system will automatically give out all the relevant python code snippets.

db-context-enrichment★ 42

A context engineering agent designed to generate, manage, and optimize structured context sets from your database schemas. It bridges the gap between Large Language Models (LLMs) and databases by compiling, evaluating, and maintaining the precise operational context needed for highly accurate natural language-to-SQL query generation.

More in Data, search & knowledge