Data pipelines & ETL
1,206 agents · most starred first
Data pipelines & ETL, page 7 of 13
A stub emulator for the Google Cloud Storage API
Storage Engine for Django Project based on Google Drive Personal Drive
A database framework for exploring genetic variation
Unified storage abstraction for Genropy framework
Dummy package for GHGSat
Dummy package for GHGSat
A Python package that uses GitHub repositories as a database for storing and retrieving data.
To use github to store files
A library that uses git version control system to store and synchronize files
Django implementation for golink
Basic S3 client with some helpful utilities
Autogenerated apitools library for storage
Google Cloud Bigquery Storage API client library
Google Cloud Storage API client library
Google Cloud Storage Control API client library
Google Cloud Storage Transfer API client library
Google Cloud Storagebatchoperations API client library
Google Cloud Storageinsights API client library
Redis storage for GottWall statistics aggregator
data-dictionary is a Python package that automates the creation of data dictionaries utilizing LLM. It generates human-readable variable names and provides comprehensive metadata, making it an essential tool for data management, enhancing data interpretability and quality assurance.
A library for processing textual datasets with large language models.
Simplifying good practice in statistical tables.
General-purpose text classifier
Storage abstraction for SQLAlchemy models with local and S3 backends, streaming uploads, and orphan garbage collection.
Private package for pulling and merging Google Trends data
Dynamic storages based on tables instead of databases
guillotina gcloud storage support
Local FS storage support for Guillotina
s3 guillotina storage support
Gumo Storage Library
Automated tool for migrating h2oGPTe collections
An statiscian package for python & very easy version
Implementation of Simple-HashStorage.
Client for the HASTE storage plaform: http://haste.research.it.uu.se
File storage for HawkAPI — local, S3, GCS, Azure backends, pre-signed URLs, streaming uploads
a storage class wrapping a dict.
a service manages storage information details
Hugging Face dataset-backed cloud file storage library
package for tree like structure sorted by category with named data
basic storage library for entities and entity definitions
Hopeit Engine Azure Blob Storage Toolkit
Hopeit Engine File System Storage Toolkit
Hopeit Engine Redis Storage Toolkit
Python library providing shared code for IDS datasets
IMAS Data Dictionaries converted to Python Dataclass used in the IMAS-GK/IDSPy parser
Tools to manipulate the IDSPy Dictionaries
Use your Email-account as Storage for data structures and files
Incremental Storage
A collection of django storage backends.
S3 storage backend for Indico
XRootD/EOS storage backend for Indico
A high-level library for traversing the OS storage model.
Ingest different data sources into structured data for LLM
Функционал получения актуального списка иноагентов с сайта МинЮста РФ
PyFS File Storage For Invenio with support for multiple checksum algos
Make blob storage group readable
Yet another iterator utils lib
ZC Buildout recipe for defining a blob-storage
An agent for collecting data for jellyfish
JSON Storage Manager for JSON data stored in text-files
JSON Storage is a simple lightweight data storage using JSON format in Python.
ERP5 Storage for jupyter
Python SDK for JustStorage object storage service
Client for Keboola Storage API
A simple library that allows the JSON storage of one key to multiple values.
A storage package for kilimall lite
Quickly get coverage statistics given reads and an assembly
An integration package connecting Airbyte and LangChain
LangChain integration for Apache Iceberg with native PyIceberg API support
An integration package connecting LakeFS and LangChain
LangChain toolkit for Microsoft Fabric OneLake — workspace, file, table, directory, and item operations powered by the Fabric MCP Server.
A dict like storage backed by a yaml file or similar method
Python3 library containing plugins for llm-dataset-converter specific to OpenAI.
Better ORM wrapper of leancloud storage python sdk.
Add your description here
Lineagentic-flow is agentic ai approach for building data lineage across diverse data processing scripts including python, sql, java, airflow, spark, etc.
Async file storage abstraction library with optional Litestar integration
BullMQ for Python (LitHaxor fork) — adds JobScheduler, global rate limit/concurrency, per-job cancellation, and QueueEvents streams.
A Django storage backend that names files by hash value.
Django Aliyun OSS (Object Storage Service) storage
Backup and restore utilities for LlamaAI data and configurations.
A package to process addresses and filter out noise
Llama CLI fetches and preprocesses learning data
llama-index Ray ingestion pipeline
llama-index packs diff private simple dataset
llama-index packs llama_dataset_metadata integration
llama-index packs RAFT Dataset paper implementation
llama-index readers airbyte_cdk integration
llama-index readers iceberg integration
llama-index readers pathway integration
llama-index readers structured_data integration
Data processing pipeline using MLX (scraper, chunker, extractor).
Data processing pipeline using MLX (scraper, chunker, extractor).
Next-Gen Hybrid Python/Rust Data Platform with MLX
LLAMASS is a Loader for the AMASS dataset
An easy-to-extend LLM annotator for robust, resumable data annotation.
Batch-oriented LLM annotation workflows for tabular datasets with OpenAI Batch support.