Data pipelines & ETL
1,206 agents · most starred first
Data pipelines & ETL, page 12 of 13
Reproducible synthetic test data for CSV and database workflows
Validate downloaded partner CSV/Parquet feeds and monitor rolling receipt deadlines
Store time series data as HDF5 files and access them through an API.
Provider package apache-airflow-providers-anthropic for Apache Airflow
Python client for Data Automation for Amazon Bedrock
Agentic Data Quality Pipeline: profiles CSVs, proposes DQ rules, writes+validates SQL to check them, and reports confirmed violations.
Lightweight database query capture agent for DBPulse — real-time monitoring for PostgreSQL, MySQL, and MongoDB
S3-compatible object storage abstraction for DEML Platform (RustFS, S3)
Production-grade DSPy library for explainable HS code classification.
A zero-boilerplate key-value storage library powered by SQLite.
A Google Drive Container storage and synchronization manager with persistent OAuth, deduplication, and verification.
Smart storage engine with data-type routing, multi-tenancy, and schema enforcement
A practical PRAGMA model implementation for tokenized tabular event histories
ForecastAgent 1.0: Zero-shot and fine-tuned time series forecasting SDK & CLI wrapper
A library for working with a directory with KDK's
Localstorage for python with more tools
Research-data storage triage: read-only stat-only scan + referenced-file-aware rotation of versioned build artifacts (e.g. SIF images)
PyPI storage-text-block-local Python Package owned by Circlez.ai
Async storage router across Redis, Upstash, PostgreSQL, and future backends
A simple text-based storage system where you can create tables, store data, and more! Encryption is included for data files.
Clean messy CSVs with an agentic workflow: an LLM decides the steps, tested code does the work.
LangChain tools for generating privacy-safe synthetic financial data for Latin American fintech applications.
Zero-shot decision tree induction and embedding using large language models
Tools to query and consume Open Knowledge Format (OKF) data catalogs, usable standalone or integrated with AI agent frameworks.
S3-like file storage using messaging platforms
Django storage backends and auto-configuration for Railway bucket storage.
A robust imputation plugin for Python Polars DataFrames backed by Rust.
Agent Data Gateway — 让 Agent 获得可信、完整、可追溯的数据输入(摄取编排层,不是 parser)
Python client for Runtime for Amazon Bedrock Data Automation
dbt-native Snowflake Cortex Agent materialization with skill and evaluation tooling
ClickHouse adapter for Meridian V1 evidence and time-series operations
Meridian V1 in-process runtime, binding resolver, and adapter SPI
Apache Kafka engine adapter for the Meridian Streaming Catalog
Provider-neutral Meridian Object Catalog contracts, streaming utilities, and conformance
PostgreSQL and PostGIS adapter for Meridian V1
Meridian V1 projection, cache, migration, and portability primitives
S3 object adapter for the Meridian V1 Object Catalog
Meridian V1 logical schema, resource, data, and metadata semantics
Provider-neutral Meridian V1 Streaming Catalog contracts and conformance suite
Allows to use server environment with storage backend
An LLM agent that standardizes legacy biomedical metadata to adhere to CEDAR templates
FshStorage API Wrapper written in Python
Allows to use server environment with storage backend FTP
Allows to use server environment with storage backend SFTP
Allows to use server environment with storage file
Stellarmesh 对象存储控制面与预签名数据面客户端
Provider-agnostic contracts for Storage Kernel.
Shared Storage Kernel implementation: paths, streams, observability and error normalization.
Dynamic provider factory for Storage Kernel.
AWS S3 provider package for Storage Kernel.
GCP Storage provider package for Storage Kernel.
Local filesystem provider for Storage Kernel.
SFTP provider package for Storage Kernel.
Extensible data structures over pluggable ordered storage backends
Scan a drive, get a ranked inventory of what fills it, classified re-fetchable or not, diffed against last time -- before you delete anything.
Provider-neutral compute transfer client for Central Storage Platform
A short description of my modern Python project
Simple and reusable S3 file upload/delete utilities for Python applications.
Database compression system evolving per-column encoding recipes and a molecular (DNA) storage layer.
Build Databricks Apps faster with a Node.js + React SDK. Built for humans and AI.
Build Databricks Apps faster with a Node.js + React SDK. Built for humans and AI.
Configurable LLM annotation with adaptive batching and BOHB-style tuning.
Browse a GTFS feed locally: cross-referenced tables, filters and a map.
Unified blob storage abstraction for Quadkit Framework - S3, GCS, Azure Blob, and local filesystem
基于 Boto3 与 aioboto3 的进程内对象存储客户端
Agentic ETL for Python: YAML pipelines where each step is a connector, a function or an LLM agent.
No-code synthetic LLM training dataset generator
A storage provider which can fetch and store media in Amazon S3.
Spoof chunked array formats (zarr, n5, neuroglancer precomputed) over HTTP with on-the-fly processing, caching, and pluggable ops.
Governed analytics and federated learning across data sources that cannot be pooled: a local agent per source, a global agent, and typed OCaml planners.
Dependency-light CLI for data engineers: profile data, run read-only SQL, read schemas, and ask a local or cloud LLM. Build pipelines with the SDLC mode.
Data API and CLI for Swarm VolumeRegistry deployments
cloudpathlib storage for fake.py
Export & back up Google Gemini chat history to JSON + Markdown, with a corruption verifier
Load, convert to Parquet and query a GTFS feed with DuckDB. The library behind GTFS Garage.
Explore your data lake without leaving Jupyter: built for data professionals
Azure Data Lake Storage support for Storage Explorer
Databricks support for Storage Explorer
Adds Jupyter Storage Explorer to the JupyterLab launcher, opening it in a tab
pandas support for Storage Explorer
Polars support for Storage Explorer
PySpark support for Storage Explorer
Machine-readable SQLite schemas and data dictionaries for 430+ web platforms.
Production-grade, modular dataset sanitization and curation for LLM training.
Mascope's File Agent: watches a folder on an instrument machine and uploads new data files to a Mascope server.
Shared object-storage providers for the potluckdb SDK
Fast synchronous key-value storage for pyxpo apps (react-native-mmkv shape, AsyncStorage method names).
RAGE Data Science & ML Toolkit for fast cleaning, EDA, preprocessing, and evaluation.
An easy to use package that encrypts and serializes data including cross-file capabilities.
Lakehouse domain pack for agent-fabric: source_inspect, medallion_plan, airflow_dag_render, dag_check
PostgreSQL and object-storage implementations of the AdCP Reliable Reporting stores
Простая библиотека для имитации localStorage в Python с сохранением в JSON
LangChain tools for official EU, Finnish and Norwegian open data (TED tenders, EU sanctions, VIES VAT, EU VAT rates, ECB FX, PRH/Brreg companies, Fingrid, FMI, Digitraffic) via hosted Apify actors, with source attribution on every record.
Owned attestation identity database tables and migrations.
Owned audit identity database tables and migrations.
Owned authentication identity database tables and migrations.
Owned authority-graph identity database tables and migrations.
Owned authorization-policy identity database tables and migrations.
Owned certificate identity database tables and migrations.
Owned scim identity database tables and migrations.