Data pipelines & ETL
1,206 agents · most starred first
Data pipelines & ETL, page 2 of 13
Apache Camel-style integration framework for .NET. 30+ EIP patterns, 29 transports (Kafka, RabbitMQ, MQTT, S3, gRPC, SFTP, SQL...), fluent C# DSL.
AMT2ABC is an open-source industrial software compiler framework that transforms industrial mechanisms into reusable Atomic Business Capabilities. It models industrial knowledge through AMT and SECP, builds AMT Graphs, and compiles business goals into ABC combinations for apps, agents, and industrial operation loops.
ADP is an intelligent data platform that bridges the gap between heterogeneous data sources and AI agents. It abstracts data complexity through business knowledge networks, provides unified data access (VEGA), and orchestrates data processing through DataFlow pipelines.
Data Curation Tool is a local-first dataset curation, tagging, model-management, downloader, 3D generation, media metadata, and tool-orchestration application. The current documentation set is stored in docs/wiki/
因果Harness:自动编排从数据处理、因果发现到交互式因果图与分析报告的完整流程,融合 LangGraph、MCP 与 RAG。AI Harness for automated causal analysis—from data preparation and causal discovery to interactive causal graphs and reports. Powered by LangGraph, MCP & RAG.
Your Platform for Text Mining through Configurable LLM Chains. Ideal for Developers and Semi-Technical Users
电商全域数据治理与 Multi-Agent 分析中台:基于 n8n + LLM 架构,实现多源电商数据 ETL、规范化数仓分层建模 (DWD/ADS) 及飞书端智能对话查询闭环。
Data science and Machine Learning AI agent
The AI-native control plane for your data stack.
High-performance and efficient Framework and Agent for creating data pipelines. The core of pipeline descriptions is based on the Configuration As Code concept and the Pkl configuration language by Apple.
Local-first Python framework for generating, executing, validating, and packaging agent training data with executable state, tool evidence, lineage, replay checks, and release artifacts.
End-to-End ML Pipeline + LLM Agent Workflow Implementation.
A trusted data asset platform for government and organizational collaboration, covering the full lifecycle of data governance, trading, authorization, analytics, and AI-powered exploration.
This repository `II-Commons` contains tools for managing text and image datasets, including loading, fetching, and embedding large datasets.
Turn documents into high-quality instruction datasets with grounding, quality filtering, deduplication, and provenance.
Let ChatGPT (Large Language Models) Serve As Data Annotator and Zero-shot/few-shot Information Extractor.
Vendor-neutral Go CLI for the Open Knowledge Format. Create, validate, lint, index, search, and graph OKF bundles. Built to be driven by AI agents.
Integrates LLMs as PTransform in Apache Beam pipelines using LangChain
A framework to do multi-threading in T-SQL using SQL Server Agent jobs.
Scrub Data: A Framework for Reproducible Data Curation with AI Coding Agents
Geopolitical news platform on GDELT: 30M+ events, about 100K added daily. Polars, Dagster and Great Expectations load MotherDuck hourly, dbt builds tested marts weekly; AI chat answers with RAG and SQL. Runs at $0/month.
Reference Java & Spring implementation of the Open Ingestion Standard (OIS). It provides a secure, decoupled, and vendor-neutral enterprise data integration platform leveraging modern Java 25 features, Spring Boot, Spring AI and vector search infrastructure to orchestrate data flows
AI-assisted validation toolkit for water, wastewater, and stormwater Utility Network schema mappings.
Dria SDK is for building and executing synthetic data generation pipelines on Dria Knowledge Network.
🤖 An autonomous AI agent system that collaboratively designs, implements, and manages Apache Airflow DAGs through natural language interaction.
🚀 Open source, fast and secure django based pastebin service
Compound AI toolchain for fast and accurate entity matching, powered by LLMs.
Wurzel is an open-source Python framework for advanced ETL pipelines in Retrieval-Augmented Generation (RAG) systems.
proof of concept to generate Airbyte low-code YAML connectors from API documentation
In this project, we implemented a topic detection system on Twitter. This system reads tweets from a data stream and assigns them to one of the existing clusters or a new one. Each cluster acts as an agent, which makes the proposed approach a multi-agent system. There is also a coordinator, who monitors the whole system and coordinates the agent.
Real-time data pipeline for AI apps in Azure
A multi-agent data wrangler project for data profiling, transformation, and quality scoring
A toolkit for biomedical data integration and harmonization
Zeltrin (formerly known as SOP, Scalable Objects Persistence)
Joltrin (formerly known as SOP, Scalable Objects Persistence)
Your own personal data engineer for OpenClaw.
本项目收集目前对话系统论文中,已公开的,用于训练中(英)文的训练集。Datasets for training Dialog.
Data pipelines and notebooks for RAG tuning using Fondant
Emotional First Aid Raw Dataset, 心理咨询问答原始语料库
Профессиональные Data Engineering Agent Skills для разработки AI Agentic Data Platform
MASSW is a comprehensive text dataset on Multi-Aspect Summarization of Scientific Workflows. MASSW includes more than 152,000 peer-reviewed publications from 17 leading computer science conferences spanning the past 50 years.
A collection of example projects demonstrating the capabilities of Fenic
🐙 AI powered CLI tool to generate mock data for anything!
AI agent for enhancing datasets with information from the internet
This project implements an agentic system for predictive maintenance using MongoDB, AWS Bedrock, LangGraph.js and Next.js.
Frags is an advanced AI agent for complex data workflows—retrieval, transformation, extraction, and aggregation. Highly customizable and extensible, it prioritizes precision.
OpenAI-powered JSON data generator.
"Sniffs out bad data"
The project demonstrates an example of how to use a supervised learning task using GPT-3.5 with JSON export, evaluating reviews in different languages
This project aims to perform sentiment analysis on tweets related to ChatGPT, a popular language model developed by OpenAI. The dataset used for training and testing consists of 219,293 tweets collected over a month. Each tweet is classified as positive ("good"), negative ("bad"), or ("neutral").
Google Cloud Cortex Framework v7 provides solution accelerators that help you build trusted, high-quality data products ready for advanced analytics and agentic use cases. The framework uses Google Cloud's products including BigQuery, Dataform, Knowledge Catalog, and Gemini Enterprise Agent Platform.
AI-driven CLI for documenting database schemas. DB + docs + codebase agents, 10 backends, BYO LLM, human-in-the-loop review.
LangChain Expression Language (LCEL) As NiFi Processors
A Baseline for an Omnipotent AI Data Catalog
Filter dialog data with a simple entropy-based method (see ACL paper)
End-to-end ELT pipeline for 160K+ Skytrax airline reviews: Airflow orchestration, BeautifulSoup scraping, S3 staging, Snowflake warehouse, dbt star schema transformation, Terraform IaC, GitHub Actions CI/CD
Retrieval augmented generation for building datasets from scientific literature: Contains the notebooks used for creating datasets
The Data Control Plane for AI Agents — acquire, validate, and land data over MCP with scoped keys, approval gates, and an audit log. Self-hosted, open source.
Agentic Data Ingestion is an intelligent ETL pipeline automation system that uses AI to generate, test, and deploy data transformation code. It combines AWS Bedrock, Strands Agents, and Model Context Protocol (MCP) to create a fully automated, human-supervised ETL pipeline generation system.
融合数据平台
ETL for RAG. Transform any source into LLM-ready markdown. Focus on your AI, not integrations.
一站式轻量化数据引擎:连接设备,交付价值;CRose是一个专为制造业与现代农业打造的集成化数据底座。它封装了从底层的协议采集(Modbus/MQTT)到上层的统计分析、UI展示的全链路能力。CRose is an integrated data platform designed for manufacturing and modern agriculture. It encapsulates the full stack capabilities from underlying protocol collection (Modbus/MQTT) to upper-level statistical analysis and UI visualizat
YAML-first, domain-driven data governance for AI agents — teach agents your business domains, metrics, and rules before they write SQL
Agentic -powered synthetic data generator that creates flowcharts using LangGraph orchestration. Features FastAPI integration, OpenAI LLM processing, and automated Mermaid diagram generation.
AI-powered pharmaceutical supply chain management using LangGraph agents, OpenAI GPT-4, and optimization algorithms.
The notebook for AI agents. Write Python. Get Markdown reports.
Sync your team's data to your LLM applications in real-time
Negochat Corpus - a dialogue corpus in the negotiation domain
Agent-interpretable data-science tools, evolved via autoresearch.
Turn any MongoDB Atlas collection into an AI-powered intelligence layer. Config-driven, domain-agnostic document enrichment powered by AgentField.
Audit a dbt project for what an AI agent will get wrong if you point it at the data today.
【2026年字节跳动AI全栈挑战赛】MarkUp 马克派 - 数据标注平台
11 specialised AI agents that autonomously clean, anonymise, validate, transform, detect anomalies and summarise any CSV, PDF or database in real time. Router Engine cuts cost by 70% and latency by 63%.
Self-hosted, AI-agent-native governed data platform with ontology, policy controls, lineage, and auditable writebacks.
📦 Prepare selected marimo notebook states once, then consume verified exports from Python, TypeScript, agents, or custom applications
Data Engineering Agent Using Open AI Function Call
This repository provides a data pipeline that syncs GitHub repositories with a free MySQL-compatible cloud database, TiDB Cloud. It can be used as a standalone data pipeline or as a personal dashboard.
Structured data & semantic tagging pipeline. Turns raw text (conversations, web pages, surveys) into tagged data for AI and search. Coordinators set ground truth; workers run LLM inference on windows. Scoring via cosine similarity. Python, FastAPI, OpenAI/Anthropic/OpenRouter, embeddings, Docker.
Code for the USDA Branded Food Dataset pipeline and the USDA Food Assistant. This project consolidates USDA FoodData Central data into a structured dataset, along with an interactive tool that allows for conversational exploration of food items, nutrients, and ingredients.
Frags is an advanced AI agent for complex data workflows—retrieval, transformation, extraction, and aggregation. Highly customizable and extensible, it prioritizes precision.
A schema-driven database that stores data as Markdown files. YAML front matter, SQL views, file watching, typed Rust codegen, and a CLI for humans and AI agents.
A shared data backend for AI agents and authorized teams.
Open source AI Agent skills for practical data development workflows
55 production-grade Google Cloud data engineering projects in Python — BigQuery, Dataflow/Apache Beam, Composer/Airflow, Pub/Sub, Dataproc, Vertex AI, Gemini, RAG, MCP. Live-tested on GCP.
Backend of cinemattr.ca
Schema-first AI analysis CLI that transforms messy data into structured insights. Define your output format, get guaranteed JSON results from any source. Combines OpenAI models with multi-tool orchestration (Code Interpreter, File Search, Web Search, MCP) for AI-powered data synthesis.
Complete open-source data platform with Airbyte, Dremio, dbt, and Apache Superset - Documented in 18 languages
A Python library for generating high-quality question-answer pairs from PDF, DOCX, MD, and TXT files
Automatically organize and classify your browser bookmarks using AI.
Intelligent, self-healing polyglot pipelines. Declarative. Observable. Autonomous.
A version-controlled lakebase for the next generation of AI applications.
AI Recommendation System & Data Engineering Lakehouse processing 21M+ records. Bridges distributed Databricks PySpark Medallion pipelines (Delta Lake SCD-2) with a 6-model PyTorch deep learning recommendation ensemble, 10-shard Neon serverless vector search (pgvector), sub-10ms serving, and an interactive cinema portal.
GenPark Distilled Skill: wukong-data-synthesizer
Autonomous Agent Count-Min Sketch Probabilistic Sublinear Heavy Hitters Frequency Estimator
Autonomous Agent Canonical Huffman Prefix Code Generator & Optimal Bitstream Serialization Engine
🧹 Automated Filtering of Undesirable Web Data to Update LLM Knowledge
Deploy autonomous AI agents to research, debug, and iterate on Kaggle data science competitions.
An AI ETL tool you drive with natural language.
Postgres test DB cloned straight from your production, for AI agents and humans. Real-time replication of rows and schema, copy-on-write clones, snapshots. No seeding, no dumps.