Data pipelines & ETL
Data pipelines and ETL projects move and reshape data: pulling records from APIs and databases, cleaning and transforming them, and keeping datasets versioned and traceable. Airbyte connects sources to warehouses and AI applications, while Dolt applies Git-style branching and history to tabular data.
1,206 agents · most starred first
Top 100 of 1,206
Dolt – Git for Data
Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.
Incremental engine for long horizon agents 🌟 Star if you like it!
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
A Python library for anomaly detection across tabular, time series, graph, text, image, and audio data. 60+ detectors, benchmark-backed ADEngine orchestration, and an agentic workflow for AI agents.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
The Open Source Feature Store for AI/ML
A system for agentic LLM-powered data processing and ETL
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
🎨 NeMo Data Designer: Generate high-quality synthetic data from scratch or from seed data.
🌀 AI-native framework for building data portals. Scaffold a full portal from a brief and load datasets in minutes with agentic skills — any backend (CKAN, GitHub, Frictionless).
Label, clean and enrich text datasets with LLMs.
Collect, aggregate, and visualize a data ecosystem's metadata
The fastest unified streaming SQL Engine in a single C++ binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute stateful context from streams, logs, metrics, events, and CDC.
A Python package for Bayesian forecasting with object-oriented design and probabilistic models under the hood.
Optimized Agentic and LLM Bulk Processing Over Your Data
Trench — Open-Source Analytics Infrastructure. A single production-ready Docker image built on ClickHouse, Kafka, and Node.js for tracking events. Easily build product analytics dashboards, LLM RAGs, observability platforms, or any other analytics product.
The backend agents build with - Multimodal database, orchestration, and serving in one file
Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 437 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.
Apache Iceberg REST Catalog in Rust — access control, credential vending and audit for every engine and AI agent. Apache 2.0.
Calling Python functions from the Ruby language
A sleek dataset viewer built entirely by AI Agent. Supports streaming large files from WebDAV, S3, SSH, Local or Hugging Face.
A backend for web, mobile, AI and IoT apps. Built on MongoDB
🎯🗯 Dataset generation for AI chatbots, NLP tasks, named entity recognition or text classification models using a simple DSL!
Data API Framework for AI Agents and Data Apps
Semantic DataFrames for humans and agents
TimeCopilot: the Agentic Forecasting Framework. Built on LLMs and Time Series Foundation Models, it lets you forecast, cross-validate, and detect anomalies using multiple foundation models through a single API. From finance and energy to web analytics, TimeCopilot turns natural-language queries into production-ready forecasts.
CRDT-based offline-first sync for SQLite. Syncs automatically with SQLite Cloud, PostgreSQL, and Supabase. No conflicts, no data loss, no backend to build. For offline-first apps and AI agents.
AI agent tooling for data engineering workflows.
potato: the portable annotation tool
[ICLR 2026] DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
Open-source data management for multimodal AI. Query, trace, and govern with a lineage-native, format-agnostic lakehouse for agents and teams. Supports biological formats and registries - by the creators of Scanpy. 🍊YC S22
🔥🔥DatalinkX异构数据源之间的数据同步系统,支持海量数据的增量或全量同步,同时支持HTTP、Oracle、MySQL、ES等数据源之间的数据流转,支持中间transform算子如SQL算子、大模型算子,底层依赖Flink、Seatunnel引擎,提供流转任务管理、任务级联配置、任务日志采集等功能🔥🔥
DataMate is an enterprise-level data processing platform designed for model fine-tuning and RAG retrieval.
Automatically reformat any JSON into any schema with AI
A Dataset for Multi-Turn Dialogue Reasoning
✨ Bootstrap annotation with zero- & few-shot learning via OpenAI GPT-3
A tool for building feature stores.
A powerful dataset generator for Rasa NLU, inspired by Chatito
A command-line interface to generate textual and conversational datasets with LLMs.
AI-native data platform: 60+ connectors, dlt-based ETL, DAG scheduling, per-source RAG knowledge bases, and sandboxed AI agents that query, chart, and analyze your data.
A distributed Spark/Scala implementation of the isolation forest and extended isolation forest algorithms for unsupervised outlier detection, featuring support for scalable training and ONNX export for easy cross-platform inference.
Implementation of statistical models to analyze time lagged conversions
LangGraph Studio template for creating an agent that does web research to genearte or enrich structured data.
PTT 八卦版問答中文語料
Generate question/answer training pairs out of raw text.
Agentic data engineering for operational data: real-time pipelines, data APIs, and data products on Kafka, Flink, Postgres, and Iceberg.
Pydantic extension for annotating autocorrecting fields.
🤖 AI Agent-driven Kaggle competition workflow. Battle-tested patterns for score stabilization, submission troubleshooting, kernel workflows, and spec-driven development.
Data ingestion service for the Agent Landing Zone. Processes PDFs, images, spreadsheets, transcripts and SharePoint content into grounding knowledge for agents, with smart chunking and multimodal embeddings.
AI-powered synthetic data generation — structured tables, unstructured documents, multi-provider LLM support, referential integrity, and code-gen mode for millions of rows
Synthetic Data Generation Toolkit for LLMs
Git For database Systems
Open source release of Data Science Agents that can analyze and transform datasets as well as train and infer models to augment your data scientists.
Real-time decision features without streaming infra. Turn live events into product reflexes — no Kafka, no Flink, no feature store.
🚀 Advanced Data & AI Engineering Portfolio: Real-world projects and production-ready patterns to level up your AI skills—from building clean data pipelines to deploying RAG systems, AI agents, and intelligent dashboards.
Natural Language Data Augmentation Tool for Conversational Systems
🔬 A Researcher&Agent-Friendly Framework for Time Series Analysis. Train Any Model on Any Dataset!
Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications
A multi-agent framework to fully automate anomaly detection in different modalities, tabular, graph, time series, and more (work in progress)!
mloda.ai - Open Data Access for AI and ML. Plugin-based. Traceable. Framework-agnostic.
AutoMind: Adaptive Knowledgeable Agent for Automated Data Science
🌟DataTonic : A Data-Capable AGI-style Agent Builder of Agents , that creates swarms , runs commands and securely processes and creates datasets, databases, visualisations, and analyses.
agent skills for autonomous data labeling, winner at openai codex hackathon 2026
Production-ready PySpark ETL template for Databricks — medallion architecture, DABs, tests, DQX, CI/CD, and agentic development with Claude Code.
🚀 Kaggle Problem Solver: AI-powered ML challenge automation Plan, code, and execute Kaggle solutions with intelligent agents. Boost your data science workflow! 🧠📊
Framework for building data agent workflows
Build polished interactive maps and report-ready figures from spatial data with AI agents.
An AI-powered process mining suite that converts natural language to process models and uses autonomous agents to analyze event logs and generate actionable process insights.
Build a dialog dataset from online books in many languages
API Framework heavily relying on the power of DuckDB and DuckDB extensions. Ready to build performant and cost-efficient APIs on top of BigQuery or Snowflake for AI Agents and Data Apps
Agentic & No-Code Data Transformations
Find which script, notebook, data, command, or AI agent produced a file — locally, with evidence and honest uncertainty.
Additive Groves, Bagged Trees with Feature Evaluation, Interaction Detection, Visualization of Feature Effects.
The AI-native wire format for structured data. 100% comprehension on every frontier model. 50-92% fewer tokens than JSON. 43B+ lossless round-trips across 17 formats. Spec v3.5.1 Stable.
🤖 Your Intelligent Copilot for Data Exploration and Processing Pipeline
Oracle AI Data Platform Workbench Samples
A data-centric AI package for ML/AI. Get the best high-quality data for the best results. Discord: https://discord.gg/t6ADqBKrdZ
SynthGenAI - Package for Generating Synthetic Datasets using LLMs.
Ultimate AWS Data & AI Platform: Real-time flight delay predictions with complete DE, DS, MLOps, Web App & Multi-Agent LLM - All deployed via CDK self-mutating pipelines
Annotation meets Large Language Models (ChatGPT, GPT-3 and alike).
DeepSeek Harness plugin: deterministic data profiling, cleaning, and verification (dsh-data-quality)
Synthetic Role-Play Conversation Dataset Generation
SmartLabelBench 2026: LLM-Powered Auto Annotation and Dataset Builder for Everything
A resource to create a multi domain Dialog Act Tagger for conversational agents using publicly available data
Task flows let you describe your problem and have an AI agent generate your Microsoft Fabric solution—workspaces, pipelines, notebooks, data agents, models, reports, and more—delivered as a CI/CD‑ready script, ready to deploy.
Versioned DuckDB databases, locally or replicated through S3, for people and agents
Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
An Agentic Data Preparation Framework for AGI-driven Scientific Discovery
A key-value database for object storage.
An opinionated Laravel package that extends FakerPHP and uses openai-php/laravel to generate fake data
worker agent API for processing data for Menome Knowledge Vault
Revolutionary AI ETL with Medallion Architecture: Zero-touch autonomous & HITL pipelines on Databricks
Agentic data engineering built on Pub/Sub for Tables. Discover, publish, transform and consume data as versioned tables with lineage, with an AI agent doing the assembly and the engineer approving.
Codify enables data scientists to perform all the tedious and time-consuming tasks such as EDA (exploratory data analysis), data cleaning, data pre-processing, data visualization, modeling, and evaluation in the data-science life cycle, by only conveying the logic of the task in natural language (English) and the system will automatically give out all the relevant python code snippets.
A context engineering agent designed to generate, manage, and optimize structured context sets from your database schemas. It bridges the gap between Large Language Models (LLMs) and databases by compiling, evaluating, and maintaining the precise operational context needed for highly accurate natural language-to-SQL query generation.