Data, search & knowledge

Data pipelines & ETL

1,206 agents · most starred first

Data pipelines & ETL, page 3 of 13

genpark-dgim-sliding-window-bit-counter-skill★ 7

Datar-Gionis-Indyk-Motwani (DGIM) streaming algorithm for estimating frequency and bit counts over massive sliding windows in logarithmic memory.

genpark-erasure-coding-reed-solomon-skill★ 7

Reed-Solomon (k+m) systematic erasure coding kernel over Galois Fields GF(2^8) tolerating up to m disk or network shard dropouts.

genpark-aries-wal-crash-recovery-engine-skill★ 7

Production-grade genpark-aries-wal-crash-recovery-engine-skill skill for AI agents

genpark-two-phase-commit-sink-exactly-once-skill★ 7

Transactional Two-Phase Commit (2PC) stream sink coordinator guaranteeing end-to-end exactly-once delivery across failure recoveries.

genpark-tumbling-sliding-session-windowing-skill★ 7

High-throughput stream window processor supporting tumbling, sliding, and dynamic inactivity-gap session windows over out-of-order event streams.

genpark-content-defined-chunking-cdc-skill★ 7

FastCDC content-defined chunking (CDC) deduplication engine with rolling hash boundary detection and min/max chunk size limits.

genpark-reservoir-sampling-algorithm-r-skill★ 7

Reservoir Sampling Algorithm R selecting a uniform random sample of k items from an unbounded, single-pass data stream in O(N).

genpark-b-link-tree-concurrent-indexing-skill★ 7

Lehman-Yao B-link tree concurrent index enabling lock-free read operations across dynamic node splits with high-key right links.

genpark-mvcc-snapshot-isolation-storage-skill★ 7

Multi-Version Concurrency Control (MVCC) engine guaranteeing non-blocking repeatable reads, snapshot isolation, and version chain garbage collection.

genpark-geohash-spatial-quadtree-encoder-skill★ 7

Production-grade genpark-geohash-spatial-quadtree-encoder-skill skill for AI agents

genpark-tdigest-streaming-quantile-sketch-skill★ 7

Production-grade genpark-tdigest-streaming-quantile-sketch-skill skill for AI agents

genpark-flajolet-martin-cardinality-sketch-skill★ 7

Flajolet-Martin (FM) distinct elements streaming sketch estimating unique cardinality from trailing zero bit patterns with logarithmic memory.

genpark-enterprise-data-lineage-impact-graph-analyzer-skill★ 7

Enterprise Data Lineage & Schema Change Blast-Radius Impact Analyzer. Constructs column- and table-level Directed Acyclic Graphs (DAGs) across Tencent Cloud Data Warehouses, Tencent Docs spreadsheets, and BI executive dashboards, performs u | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550

polynx★ 7

pandas-style query/eval/assign for Polars: string expressions compiled to native Polars expressions. Eager + lazy, @var, UDFs, agent-ready.

genpark-quic-multiplexed-stream-framing-skill★ 7

Production-grade genpark-quic-multiplexed-stream-framing-skill skill for AI agents

llm-product-insights★ 6

Extracting Product Insights from Unstructured text data using LLMs with LangChain

llm-dataset-generator★ 6

The LLM Dataset Generator is an open source tool for generating text data compatible with various language models supported by LangChain. You can customize it to meet your specific needs, making it a valuable resource for researchers, developers, and organizations working on NLP applications.

resilient-rap-framework★ 6

A resilient, fault‑tolerant telemetry analytics pipeline designed to validate, benchmark, and stress‑test high‑frequency sensor data streams under real‑world failure conditions. Includes chaos testing, DLQ repair, GPU‑accelerated ingestion, and end‑to‑end reliability validation for motorsport‑grade telemetry environments.

dbt_unified_rag★ 6

Fivetran dbt package designed to generate an end model and Cortex Search Service (for Snowflake destinations only) which contains unstructured document data to be used for Retrieval Augmented Generation (RAG) applications leveraging Large Language Models (LLMs)

sentiment-extraction-with-llms★ 6
aegis-dq★ 6

Open, audit-grade agentic data quality framework with portable industry packs

Factory-Intelligence-Platform★ 6

An open-source factory intelligence platform for quality drift detection, synthetic factory simulation, governed AI workflows, and industrial operations intelligence.

visualize_dataset★ 6

docker region:us

agentic-ai-cold-chain-optimization★ 6

Agentic AI for Cold-Chain Sustainability: LSTM-Based Predictive Logistics

OrgSmith★ 6

Synthetic data generator for hallucinating entire companies as browsable file shares of realistic business docs & emails with a deterministic ground-truth answer key. Generates: .docx/.pdf/.xlsx/.pptx/.eml, and pre-2007 binaries. Used for augmenting research data sets and testing agentic systems against meaningful synthetic examples.

semantic-metrics-modeling-assistant★ 5

MCP agent for semantic metrics governance with built-in trust scoring, lineage visualization, and conversational metric definition. Designed for data teams working with dbt, LookML, and modern semantic layers.

dataworkers-claw-community★ 5

We’re build a swarm of agents for all data tasks. That anyone can use for free, open-source community version.

gpu-accelerated-ds-agents★ 5

GPU-Accelerated Data Science Agents

loafer★ 5

loafer is an AI agent designed to automate the ETL (Extract, Transform, Load) process, enabling data engineers to efficiently transform and load data into their target systems.

ogd_ai-analyzer★ 5

Automagically do a deep check of metadata quality of your DCAT OGD offering.

enrichment★ 5

Data enrichment with AI for pandas DataFrame

company-classification-research★ 5

This project explores the use of various NLP techniques to classify companies by analyzing their textual descriptions

nba-hate-tracker★ 5

Who is r/NBA's most hated player? Every comment of a season, classified.

RATE★ 5

RATE: An LLM-Powered Retrieval Augmented Generation Technology-Extraction Pipeline

feldera-demos★ 5

Demos showing how to build with Feldera

ondine★ 5

Batch-process tabular data with any LLM. pandas & Polars in, structured columns out. Checkpointing, budget limits, and 100+ providers built in.

rushdb-python★ 5

RushDB SDK for Python: RushDB Cloud or self-hosted RushDB.

gdelt-cli★ 5

The command-line interface for the GDELT Project — built for agents, optimized for automation.

slopcollector★ 5

🦝 SlopCollector - a developer tool for eliminating trash from your Supabase

c1v-id★ 5

Identity resolution for AI applications

motherduck-control-tower★ 5

A self-updating ops console for your whole MotherDuck account — the data-flow graph of every dive, flight, table, and share, built from the manifest comments in your own objects. Load INSTALL.md into your AI assistant to install it.

mloda-registry★ 5

Registry of official plugins for mloda.ai

dskit★ 5

Template copier para proyectos de Data Science con uv, Sphinx y estructura modular

building-data-pipelines-with-ai★ 5

Using AI to 10x Data Pipeline Dev Speed

job-webscraper★ 5

LLM API-Powered Job Listings Data Cleaning Tool: Showcase for Data Scientists

chatbot-conversation-analysis★ 5

🤖 Analyze chatbot conversations to enhance performance and user satisfaction through data analytics, clustering, and predictive modeling.

demoverse★ 5

Grow a fake company's entire sales history across CRM, calls, email and Slack that moves every week. Synthetic demo-data engine for B2B products.

citybehavex★ 5

CityBehavEx is a scalable, empirically validated, LLM-assisted urban behavior simulation platform. It generates synthetic city-scale mobility trajectories and lets users inspect, replay, debug, and evaluate them against observed mobility, time-use, semantic, transport, and social-network patterns.

odeduck★ 5

공공데이터 덕후, 오데덕. AI 에이전트가 목표에 맞는 자료를 찾고, 자동 활용신청·수집·연결·분석을 거쳐 결과와 근거를 만들도록 돕는 오픈소스 실행 플랫폼.

Cleanframe★ 5

The reproducible data-cleaning engine for Python. Profile messy CSVs, detect issues, generate a YAML recipe (with or without an LLM), and replay it with zero AI calls.

FHIR-to-OMOP★ 5

An end-to-end clinical data pipeline mapping FHIR to OMOP CDM v5.4. Features DuckDB, RAG-driven semantic mapping, Human-in-the-Loop governance, and OHDSI DQD validation.

accessclone★ 5

Convert Microsoft Access databases to modern web applications — extract the intent, not the code. AI-powered pipeline: tables, queries, forms, reports, VBA modules → PostgreSQL + browser UI with structured intents for autonomous agent operation.

hql★ 5

HQL — Hyper Query Language — is a strongly typed language for querying, transforming, connecting and computing over data, knowledge and AI workflows.

DBT_Athena_with_Kafka★ 4

Kafka Integration with Dbt

HoloChatStats★ 4

VTuber data visualization site with ETL pipeline and AI agent

Datapillar★ 4

Raw In, Golden Wings Out

export-dialogflow-logs-to-bigquery★ 4

Export Dialogflow conversation logs to BigQuery with masking PII using DLP API

llamaindex-openai-mistral-tagging★ 4

Tagging using LLM

autochain-bot★ 4

This project is dedicated to building an intelligent data processing pipeline using AutoChain LLM and BERT ML.

mycelium★ 4

A powerful platform designed to simplify the creation and management of Data Contracts, bridging systems for seamless data ingestion

Data_Pipeline_AI★ 4

Data Pipeline with validation + AI Assistant

SmartLableBench-beta_-★ 4

自动标注!支持世间万物!由llm解构提示词,再组合标注,轻松制作数据集

metadata_agent★ 4
agent-k★ 4

Autonomous multi-agent system for discovering, entering, and winning Kaggle competitions

polaris_studio★ 4

💫 Visual data pipelines with AI-reviewed transformations and live spreadsheet output, built offline-first with Polars and PySide6.

ragloom★ 4

Logstash-like RAG ingestion daemon for local files, chunking, embeddings, and Qdrant indexing.

agentic-data-platform★ 4

An agentic data platform for e-commerce analytics, combining data engineering pipelines, a serving layer, and multi-agent orchestration, with a clear path toward GenAI, MLOps, DevOps, and cloud deployment.

rag-ingestion-pipeline★ 4

An opinionated, local-first, asynchronous document ingestion pipeline for RAG workflows.

urgap★ 4

urgap - unified resource governance and data provenance

FastAPI-Kafka-MongoDB-OpenAI★ 4

Create a scalable FastAPI application that serves an endpoint to recommend things to do in a given country during a specific season by consulting the OpenAI API, store the results in MongoDB.

arche★ 4

An open-source engine for deciding when messy records of people, places, organisations, and products refer to the same real-world entity, with evidence, not just similarity scores or identifiers.

DataPilot★ 4

A Plan-and-Verify LLM Agent for Repository-Level Data Engineering

github-stars-to-markdown★ 4

一个轻量级工具,一键将 GitHub Star 导出为纯净的 Markdown 笔记。 | A lightweight tool to export GitHub Stars to clean Markdown files.

streamguard★ 4

Real-time fraud detection pipeline using Kafka, Airflow, dbt and Delta Lake, with a CNN scorer and an agentic RAG investigator (LangGraph + Qdrant), fully Dockerized. Designed and Developed by NIKHIL CHARY SRIRAMOJU

synthesis-through-simulation★ 4

[NeurIPS 2026] Generating Coherent Enterprise Data via Scalable Agent-System Interaction

kafka-connect-ai★ 4

AI-powered universal Kafka Connect connector — transform data between any source and sink using LLMs

JobLake★ 3

AI-powered lakehouse platform for IT job market analytics

rag-ready-extractor★ 3

Stop indexing noise. Turn messy websites and PDFs into clean, structured data for RAG pipelines with semantic importance scoring and token optimization.

ma-stat-dsm★ 3

A multi-agent statistical discriminative sub-trajectory mining technique for analyzing the trajectories of multiple agents using SportVU player tracking data from the 2015/16 season.

claude-mongo-exasol-pipeline★ 3

Claude Code agents move any MongoDB collection into Exasol, discover patterns, and publish a live data app — one prompt, end to end.

Refining-Spotify-Dataset-with-LLAMA3-70B★ 3

Reclassifying Spotify tracks Explicity with LLAMA3-70B built using LangChain

Autonomous_Text_Tagging_App★ 3

streamlit region:us

LakeMind★ 3

AI DataLake

sage_tree_converter★ 3

A modular, data-agnostic toolkit for converting N-body simulation merger trees from various formats (ASCII, Binary, HDF5) and tool-chains (AHF, Rockstar, Subfind, Gadget-4) into SAGE-compatible HDF5 and binary files.

adii_team★ 3

A data-incident agent whose fixes must pass an authorizer and an independent validator, and the evaluation behind the paper "Permission Is Not Justification": a frozen twin benchmark, registered runs, and a checker that recomputes every reported result from the published evidence.

AirQualityPredictor★ 2

Air quality prediction using publicly available sensor data (temperature, humidity, etc...)

TableConfigAgent★ 2

A Configurable Agentic Platform (With Hugging Face Transformers And Langchain) To Draft Semantically Tablassert TableConfigs From Natural Language Inputs.

data-wrangler★ 2

AI-Assisted Data Wrangler & Visualizer | Streamlit

YedaFlow_Palantir★ 2

Cross-system order intelligence pipeline on Palantir Foundry with cited exceptions and a governed read-only agent hand-off

puredata★ 2

puredata — Website Landing page for puredata, the Python library for automatic data cleaning and silent drift detection.

Autonomous_Data_Scientist★ 2

gradio region:us

gpt3-quality-filter★ 2

gradio region:us

nlp-gpt3-zero-shot-classification-app★ 2

gradio region:us

bias_agent★ 2

Autonomous LLM agent auditing any CSV for statistical and intersectional bias. Chi-squared, Cramer's V, LLM-as-judge. Live demo on HF Spaces.

BiliBili-DataViz-Plugin★ 2

在本地整理 B站观看历史与收藏的 Chrome 和 Edge 扩展。当前为 0.13.0-alpha 预发布。

nba-ai-ml★ 2

Analytics-engineering warehouse (dbt, DuckDB, MotherDuck) around a frozen NBA player-stat model: contracts, reconciliation to committed artifacts, a 23-feature SQL parity proof, lineage ending in exposures, and a pre-registered out-of-sample paper evaluation against market lines. Free tiers only.

almanac★ 2

An ML platform for work-queue risk, built on the GitHub public event firehose. Point-in-time-correct feature store, PySpark medallion, MLflow serving, and a bounded LLM agent whose every claim is mechanically verified against its own tool calls.

csvx★ 2

CSVX is an open, plain-text spreadsheet format for the AI era: human- and AI-readable CSV-shaped tables with types, formulas, formatting, comments, structured annotations, multi-line cells, and sheets.

file-to-database-skill★ 2

Turn data files (CSV/Excel/JSON/JSONL/TSV) into a validated database, then operate it in natural language — pluggable methodology, quality gates, migration ledger, visual delivery

sqlite-transit-sync★ 2

Local-first SQLite synchronization through verified snapshots and configurable row-level merge policies. Zero dependencies, Python 3.10+.

More in Data, search & knowledge