Data, search & knowledge

Data pipelines & ETL

1,206 agents · most starred first

Data pipelines & ETL, page 12 of 13

agent-paranoid-android

Reproducible synthetic test data for CSV and database workflows

agentsor-file

Validate downloaded partner CSV/Parquet feeds and monitor rolling receipt deadlines

aitana-storage

Store time series data as HDF5 files and access them through an API.

apache-airflow-providers-anthropic

Provider package apache-airflow-providers-anthropic for Apache Airflow

aws_sdk_bedrock_data_automation

Python client for Data Automation for Amazon Bedrock

data-quality-agent-dk

Agentic Data Quality Pipeline: profiles CSVs, proposes DQ rules, writes+validates SQL to check them, and reports confirmed violations.

dbpulse-agent

Lightweight database query capture agent for DBPulse — real-time monitoring for PostgreSQL, MySQL, and MongoDB

deml-storage

S3-compatible object storage abstraction for DEML Platform (RustFS, S3)

dspy-nomenclator

Production-grade DSPy library for explainable HS code classification.

easydb-storage

A zero-boilerplate key-value storage library powered by SQLite.

easystorage

A Google Drive Container storage and synchronization manager with persistent OAuth, deduplication, and verification.

evoid-smart-storage

Smart storage engine with data-type routing, multi-tenancy, and schema enforcement

fastpragma

A practical PRAGMA model implementation for tokenized tabular event histories

forecast-agent-sdk

ForecastAgent 1.0: Zero-shot and fine-tuned time series forecasting SDK & CLI wrapper

kdk-storage

A library for working with a directory with KDK's

localstorage

Localstorage for python with more tools

scitex-storage

Research-data storage triage: read-only stat-only scan + referenced-file-aware rotation of versioned build artifacts (e.g. SIF images)

storage-text-block-local

PyPI storage-text-block-local Python Package owned by Circlez.ai

storage-verse-avneesh

Async storage router across Redis, Upstash, PostgreSQL, and future backends

storagepy

A simple text-based storage system where you can create tables, store data, and more! Encryption is included for data files.

agentic-csv-cleaner

Clean messy CSVs with an agentic workflow: an LLM decides the steps, tested code does the work.

langchain-latam-synth

LangChain tools for generating privacy-safe synthetic financial data for Latin American fintech applications.

llm-trees

Zero-shot decision tree induction and embedding using large language models

okf-agent-tools

Tools to query and consume Open Knowledge Format (OKF) data catalogs, usable standalone or integrated with AI agent frameworks.

chatstorage3

S3-like file storage using messaging platforms

django-railway-storage

Django storage backends and auto-configuration for Railway bucket storage.

dragon-imputation

A robust imputation plugin for Python Polars DataFrames backed by Rust.

agent-data-gateway

Agent Data Gateway — 让 Agent 获得可信、完整、可追溯的数据输入(摄取编排层,不是 parser)

aws_sdk_bedrock_data_automation_runtime

Python client for Runtime for Amazon Bedrock Data Automation

dbt-cortex-agent

dbt-native Snowflake Cortex Agent materialization with skill and evaluation tooling

meridian-storage-clickhouse

ClickHouse adapter for Meridian V1 evidence and time-series operations

meridian-storage-core

Meridian V1 in-process runtime, binding resolver, and adapter SPI

meridian-storage-kafka

Apache Kafka engine adapter for the Meridian Streaming Catalog

meridian-storage-object-common

Provider-neutral Meridian Object Catalog contracts, streaming utilities, and conformance

meridian-storage-postgresql

PostgreSQL and PostGIS adapter for Meridian V1

meridian-storage-projection

Meridian V1 projection, cache, migration, and portability primitives

meridian-storage-s3

S3 object adapter for the Meridian V1 Object Catalog

meridian-storage-semantics

Meridian V1 logical schema, resource, data, and metadata semantics

meridian-storage-streaming

Provider-neutral Meridian V1 Streaming Catalog contracts and conformance suite

odoo-addon-storage-backend-environment

Allows to use server environment with storage backend

arms-agent

An LLM agent that standardizes legacy biomedical metadata to adhere to CEDAR templates

fshstorage

FshStorage API Wrapper written in Python

odoo-addon-storage-backend-ftp-environment

Allows to use server environment with storage backend FTP

odoo-addon-storage-backend-sftp-environment

Allows to use server environment with storage backend SFTP

odoo-addon-storage-file-environment

Allows to use server environment with storage file

stellarmesh-storage

Stellarmesh 对象存储控制面与预签名数据面客户端

storage-kernel-contracts

Provider-agnostic contracts for Storage Kernel.

storage-kernel-core

Shared Storage Kernel implementation: paths, streams, observability and error normalization.

storage-kernel-factory

Dynamic provider factory for Storage Kernel.

storage-kernel-provider-aws-s3

AWS S3 provider package for Storage Kernel.

storage-kernel-provider-gcp-storage

GCP Storage provider package for Storage Kernel.

storage-kernel-provider-local-filesystem

Local filesystem provider for Storage Kernel.

storage-kernel-provider-sftp

SFTP provider package for Storage Kernel.

tketool.storage

Extensible data structures over pluggable ordered storage backends

awstorage

Scan a drive, get a ranked inventory of what fills it, classified re-fetchable or not, diffed against last time -- before you delete anything.

central-storage-platform-compute

Provider-neutral compute transfer client for Central Storage Platform

personal-book-storage

A short description of my modern Python project

s3-storage

Simple and reusable S3 file upload/delete utilities for Python applications.

storage-researcher

Database compression system evolving per-column encoding recipes and a molecular (DNA) storage layer.

@databricks/appkit

Build Databricks Apps faster with a Node.js + React SDK. Built for humans and AI.

@databricks/appkit-ui

Build Databricks Apps faster with a Node.js + React SDK. Built for humans and AI.

cabsallm

Configurable LLM annotation with adaptive batching and BOHB-style tuning.

gtfs-garage

Browse a GTFS feed locally: cross-referenced tables, filters and a map.

quadkit-storage

Unified blob storage abstraction for Quadkit Framework - S3, GCS, Azure Blob, and local filesystem

stellarmesh-objectstorage

基于 Boto3 与 aioboto3 的进程内对象存储客户端

agentl-etl

Agentic ETL for Python: YAML pipelines where each step is a connector, a function or an LLM agent.

brainbrew-llm

No-code synthetic LLM training dataset generator

cgm-s3-storage-provider

A storage provider which can fetch and store media in Amazon S3.

chunkmirage

Spoof chunked array formats (zarr, n5, neuroglancer precomputed) over HTTP with on-the-fly processing, caching, and pluggable ops.

dagents

Governed analytics and federated learning across data sources that cannot be pooled: a local agent per source, a global agent, and typed OCaml planners.

de-agentic

Dependency-light CLI for data engineers: profile data, run read-only SQL, read schemas, and ask a local or cloud LLM. Build pipelines with the SDLC mode.

ethswarm-volumes

Data API and CLI for Swarm VolumeRegistry deployments

fake-py-cloudpathlib-storage

cloudpathlib storage for fake.py

gemini-chat-exporter

Export & back up Google Gemini chat history to JSON + Markdown, with a corruption verifier

gtfs-garage-core

Load, convert to Parquet and query a GTFS feed with DuckDB. The library behind GTFS Garage.

jupyter-storage-explorer

Explore your data lake without leaving Jupyter: built for data professionals

jupyter-storage-explorer-abfs-support

Azure Data Lake Storage support for Storage Explorer

jupyter-storage-explorer-databricks-support

Databricks support for Storage Explorer

jupyter-storage-explorer-launcher

Adds Jupyter Storage Explorer to the JupyterLab launcher, opening it in a tab

jupyter-storage-explorer-pandas-support

pandas support for Storage Explorer

jupyter-storage-explorer-polars-support

Polars support for Storage Explorer

jupyter-storage-explorer-pyspark-support

PySpark support for Storage Explorer

keywordbarrage-schemas

Machine-readable SQLite schemas and data dictionaries for 430+ web platforms.

llm-sanitizer-pro

Production-grade, modular dataset sanitization and curation for LLM training.

mascope_file_agent

Mascope's File Agent: watches a folder on an instrument machine and uploads new data files to a Mascope server.

potluckdb-storage

Shared object-storage providers for the potluckdb SDK

pyxpo-storage

Fast synchronous key-value storage for pyxpo apps (react-native-mmkv shape, AsyncStorage method names).

rage-tools

RAGE Data Science & ML Toolkit for fast cleaning, EDA, preprocessing, and evaluation.

simplstorage

An easy to use package that encrypts and serializes data including cross-file capabilities.

spec-agents-lakehouse

Lakehouse domain pack for agent-fabric: source_inspect, medallion_plan, airflow_dag_render, dag_check

adcp-reporting-storage

PostgreSQL and object-storage implementations of the AdCP Reliable Reporting stores

localstorage-py-local

Простая библиотека для имитации localStorage в Python с сохранением в JSON

langchain-eu-nordic-open-data

LangChain tools for official EU, Finnish and Norwegian open data (TED tenders, EU sanctions, VIES VAT, EU VAT rates, ECB FX, PRH/Brreg companies, Fingrid, FMI, Digitraffic) via hosted Apify actors, with source attribution on every record.

tba-storage-attestation

Owned attestation identity database tables and migrations.

tba-storage-audit

Owned audit identity database tables and migrations.

tba-storage-authentication

Owned authentication identity database tables and migrations.

tba-storage-authority-graph

Owned authority-graph identity database tables and migrations.

tba-storage-authorization-policy

Owned authorization-policy identity database tables and migrations.

tba-storage-certificate

Owned certificate identity database tables and migrations.

tba-storage-scim

Owned scim identity database tables and migrations.

More in Data, search & knowledge