Data pipelines & ETL
1,206 agents · most starred first
Data pipelines & ETL, page 9 of 13
Management CRUD in MongoDB
Pragmastat: Pragmatic Statistical Toolkit
Parallel random access to gzip files
Storage implementation for Django that interacts with Google Drive
Add your description here
NER training-data generation built on py-agent-lib. Bring your own labels, LLM, and downstream trainer.
Store some information within the current .py file for the distribution of a single-source-file Python application.
Parallel Python Averager for Climate Data
Python implementation of the Ball Mapper algorithm.
Fragility function fitting for any data type, with misspecification-robust uncertainty.
pyJsonStorage is a Python package designed to implement basic database functionalities using JSON (JavaScript Object Notation). This package provides various features such as creating a new database, inserting rows, updating, and deleting rows. It leverages the JSON module from Python's core libraries for managing JSON files.
python List of Dict (Table) Storage library
Simple data storages package
Beeswarm plots for python
Use your pydantic models as an ORM, storing data in Redis.
Pydantic models for Apache Airflow
A Python library for managing Airtable data using Pydantic objects
A lightweight, type-safe storage system for Pydantic models with support for multiple backends and persistent storage.
A python data storage backend library.
Python wrapper library providing LoL ddragon data asynchronously
File Storage
A python tool to extract coverage from gvcf files.
A package to store data on the hard disk (HD) and make it available to all Python applications running locally!
A simple python package to convert geo data to OpenAir format
Python Ragic API client for data loading and manipulation.
Queue storage backend interfacing with cloud providers.
Adds a filesystem-based queue extension to python-slimta.
Adds a redis-based queue extension to python-slimta.
Wechat Storage Module for Python.
Provide a variety of storage backends in a single library.
Simple local file storage backed by SQLite with optional AES-GCM encryption
Qai Storage API
Django storage power by qiniu store
A collection of storage modules like for file management, metadata and more
A regular radicale storage with an additional sqlite index on specified fields.
DecSync storage plugin for Radicale
DEPRECATED: merged into etesync-dav
Basic scheduling for users on the same server
A time range indexed radicale storage
Generate QA pairs from JSON documents to evaluate RAG pipelines
Git for data scientists - manage your code and data together
RAG dataset generator
lightweight Extract-Transform-Load (ETL) framework for Python 3+
provides an archetypes storage which stores files on the filesystem into a mercurial repository
Client for the Research Data Storage Registry, Research Computing Services, University of Melbourne.
reshape data type in py language
A small example package
Reserved package to prevent dependency confusion.
A readonly wrapper around a ZODB storage
Recc Object Storage
Django hex storage is add-on on the top of Django file system. It has improvements that cover: international filenames (processed with pinyin module) and the same file names (django adds underscore to same filename, we add HEX hash) - hence the name :)
Redis json stroage.
Packs a History Free PostgreSQL RelStorage for ZODB.
Aids users in publishing data to a Record Evolution Datapod
好!
Storage handlers
File storage utility implementation
Fast Python bindings for Project Haystack — types, codecs (Zinc/Trio/JSON), filters, ontology, and Xeto support powered by Rust
s3 using tool
S3 Storage Stats
A storage provider which can fetch and store media in Amazon S3.
S3 safe storage
A simple Storage backend for Django that makes use of Amazon's S3 service.
A comprehensive Python helper for Azure Storage Blob and Queue operations with file explorer-like functionality
LLM components for the Sayou Data Platform
Tools for working with scATAC-seq fragment files.
AI-powered dataset segmentation agent for ML workflows
Sample storage add-on for SENAITE
Parallelise version of sequana_coverage standalone application.
Pluggable file upload + storage (local or S3 via extras) module for simple_module apps
File storage made easy
A small, easy to use module that helps you store data easily.
Assist in the handling of files on storages from different providers.
Storage client for local and gcloud
SingleFile Rest Storage for python.
LLM-powered feature engineering with scikit-learn API compatibility
This package provides a stable interface for interactions between Snakemake and its storage plugins.
A Snakemake storage plugin for the LHCb apd API.
A Snakemake storage plugin to read and write from Azure Blob Storage
Snakemake storage plugin for downloading files via HTTP with caching and rate limiting
A Snakemake storage plugin for S3 API storage (AWS S3, MinIO, etc.)
A Snakemake storage plugin for the DIRAC Data Management System (DMS).
A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync
A Snakemake plugin for handling input and output via FTP
A Snakemake storage plugin for Google Cloud Storage
A Snakemake storage plugin for downloading remote Git repositories.
Snakemake storage plugin for downloading input files from HTTP(s).
A Snakemake plugin for handling input and output on iRODS
Snakemake storage plugin for downloading Git LFS files with optional local caching
Snakemake plugin to interact with the NERSC filesystem
A Snakemake Storage Plugin for Pelican Federations
A Snakemake storage plugin that handles files available through Rucio.
A Snakemake storage plugin for S3 API storage (AWS S3, MinIO, etc.)