Data, search & knowledge

Data pipelines & ETL

1,206 agents · most starred first

Data pipelines & ETL, page 4 of 13

DivvyAI-Public★ 2

Chicago Divvy bike-share availability on Google Cloud: a Cloud Run pipeline collects live data every 15 minutes into BigQuery, dbt models and checks it hourly, and a hosted AI agent (Strands + MCP) answers plain-English questions from read-only data tools. Complete code private, only shareable part published in this repo

mechon-ai★ 2

Agentic Dataset Engineering Workstation

data-cleaning-llm★ 1

streamlit region:us

DataPilot-AI-Agent★ 1

docker region:us

agentic-schema-review★ 1

docker region:us

dataqualityandanomaly-detection-agent

DataQuality&Anomaly Detection Agent review: compare features, pricing, use cases, access model, and alternatives for this Data Science agent in 2026. Ensure Data Quality, Spot Anomalies, and Validate Readiness — All in One Click

magick

Magick review: compare features, pricing, use cases, access model, and alternatives for this Coding Assistant agent in 2026. Visual AI development environment for no-code data pipelines.

tensorstax

TensorStax review: compare features, pricing, use cases, access model, and alternatives for this Data Science agent in 2026. Autonomous AI Agents for Data Engineering

Genesis Bots

Automate critical data workflows with AI agents, no coding needed.

Cognite Atlas AI

Automate complex industrial tasks using AI that truly understands your factory.

Tilores

Get a single, clear view of all your customer data in real time.

openhop

Animated data-flow diagrams your AI agent can write. CLI.

@luccapinto/agentic-data-kit

AI Agent templates - Skills, Agents, and Workflows for Data Teams

ip2location-nodejs

IP2Location geolocation component

@ruvector/agentic-synth

High-performance synthetic data generator for AI/ML training, RAG systems, and agentic workflows with DSPy.ts, Gemini, OpenRouter, and vector databases

@claudeautopm/plugin-data

Complete data engineering plugin with Airflow, Kedro, LangGraph, Kafka, dbt, pandas experts, data quality rules, and example scripts for ClaudeAutoPM

Aries-storage

Read and write files on Google Cloud Storage and Amazon S3 as if they are on local computer.

BalancedDiscStorage

Makes sure, that there are never more files in one directory, than your LIMIT.

BerkeleyDB-Backend-Storage-Engine-for-DURUS

This code provides a new storage engine for DURUS, an excellent persistence system for Python

BetterJSONStorage

An optimized tinyDB storage extension

CirclesS3Storage

PyPI Package for Circles Storage functions

Coherence

Coherence - DLNA/UPnP framework for the digital living

Data-Generation-Agents

AI-powered synthetic data generation pipeline with web search, topic extraction, and persistent state management

DataXstorage

Awesome and easy to use database

DeltaAgent

A library used to fetch data from deltalake tables locally.

DirectoryStorage

A ZODB Storage which uses one file per revision of an object

DoDoStorage

User-Space write-once Network Filesystem

Dragon-Ball-API

Api unofficial de dragon ball inspirado de la fandom de dragon ball

Flask-Azure-Storage

Flask extension that provides integration with Azure Storage

Flask-GoogleStorage

Google Cloud Storage for Flask

Flask-QiniuStorage

Qiniu Storage for Flask

Flask-QiniuStoraging

Qiniu object storage for flask

Flask-SaeStorage

SAE Storage for Flask

Flask-Storage-Helpers

Various file storage backends for Flask apps.

FragStatsPy

A Python wrapper for Fragstats.

FullMatrix

full matrix coordinates

Geometric-Particle-Swarm-Optimisation

A python package to find the best features in a dataset using Geometric Particle Swarm Optimisation

IduStorage

--/--

KassStorager

Biblioteca para gerenciamento de diretórios e arquivos

KegStorage

A simple storage interface with multiple backends for use in a Keg app.

LocalDataStorage

This Python package creates a file locally in the user's device more easily.

MT-PY-S3-STORAGE-BUCKET

A Mouritech package

Nerd-Storage

A simple LAN storage.

NetStorageKit

Akamai's NetStorage API communication kit

OpenAISurveyWrapper

wrapper for the OpenAI to label survey results

Orange3-Azure-Data-Lake-Storage-Gen2

Widgets to load and save tables to/from Azure Data Lake Storage (ADLS) Gen 2

ProTaska-GPT

Unleash the Potential of Datasets with Intelligent Tasks, Tutorials, and Algorithm Recommendations.

Products.ExternalStorage

An add-on Plone product which provides an extra storage for Archetypes.

Products.TIDStorage

TIDStorage Product provides a way to have consistent backups when running a multi-storage instance (only ZEO is supported at the moment).

PyActiveStorage

A Python client for Active Storage

PyGPTK

This repository contains Python functions for extracting gene features from a GFF3 file and performing keyword analysis using TF-IDF (Term Frequency-Inverse Document Frequency).

RelStorage

A backend for ZODB that stores pickles in a relational database.

S3CachedStorage
SDG-LLM

Synthetic data generator for LLM evaluation

StorageAlchemy

Use sqlalchemy to commit/rollback changes to files on the filesystem or any other external storage.

StorageTopology

A scientific computation library for performance analysis of erasure coding and/or replication in a distributed data storage environment.

Tshrag-MetricDB

MetricDB is a database for storing and querying time based metrics data.

UpStorageApiClient

UpStorage Api Client.

ValStorage
XDatastorage

Data handler with txt,and jsons

XcomStorage

xflow xcom storage

abc-storage

This service implements storage of abc project

abs-blob-storage-manager-core

Core utilities for managing blob storage operations with support for various cloud providers and local storage systems

adri-storage

ADRI Storage Python SDK

ae-cloud-storage

ae namespace module portion cloud_storage: distribute files to and retrieve them from cloud storage hosts

agent-as-annotators

Agent-as-Annotators: Structured Distillation of Web Agent Capabilities

agent-data-cli

Unified multi-source content CLI for discovery, sync, local query, and explicit interaction.

agent-data-toolkit

Connect agents to data

agent-data-tools

Agent 训练数据的校验、转换与批处理工具。

agentfabric

Config-driven database component for agent pipelines (PostgreSQL + SQLAlchemy + Pydantic + fsspec).

agentic-data

Universal data comprehension for AI agents

agentic-data-contracts

YAML-first, domain-driven data governance for AI agents

agentic-de

Agentic Data Engineering tools and protocols (MCP, stdio, sse, FastAPI, etc)

agentic-hdf5

HDF5 tools and semantic metadata for agentic workflows

agentic-swmm-workflow

Unified CLI for reproducible and auditable Agentic SWMM workflows.

agentprep

Intelligent ML preprocessing pipeline tool with AI agents

airbyte-agent-airtable

Airbyte Airtable Connector for AI platforms

airbyte-agent-amplitude

Airbyte Amplitude Connector for AI platforms

airbyte-agent-github

Airbyte Github Connector for AI platforms

airbyte-agent-gitlab

Airbyte Gitlab Connector for AI platforms

airbyte-agent-google-drive

Airbyte Google-Drive Connector for AI platforms

airbyte-source-azure-blob-storage

Source implementation for Azure Blob Storage.

airbyte-source-zapier-supported-storage

Source implementation for Zapier Supported Storage.

airflow-plugin-config-storage

Inject connections into the airflow database from configuration

airless-google-cloud-storage

Airless package to work with google cloud storage

alchemical-storage

Storage based off of sqlalchemy_marshmallow

alembic-autogen-check

CLI to check with alembic migrations are in sync with your SQLAlchemy models.

alembic-autogenerate-enums

Alembic hook that allows enums values to be upgraded and downgraded in migrations automatically

alembic-pg-autogen

Alembic autogenerate extension for PostgreSQL-specific objects

aleph-blobstorage

Tool for accessing blob storage

alephvault-evm-events-http-mongodb-storage

A trait to implement a standard mongodb server which logs events and also caches some data related to them

alephvault-http-mongodb-storage

A lightweight server to work as a simple storage for games, done with MongoDB and Flask

alephvault-windrose-http-mongodb-storage-generator

A generator of production-ready HTTP storage stacks for WindRose/NetRose games

alliance-platform-storage

Alliance Platform Storage

allmeans

Automatic topic discovery with zero LLMs, minimal input

allmydata-tahoe

secure, decentralized, fault-tolerant filesystem

ambition-inmemorystorage

A non-persistent in-memory data storage backend for Django.

amsdal_storages

amsdal_storages plugin for AMSDAL Framework

apache-airflow-dragdrop-plugin

A custom Apache Airflow plugin to integrate a React UI

apiqa-storage

Apiqa user storage backend for django projects

More in Data, search & knowledge