Data, search & knowledge

Data pipelines & ETL

1,206 agents · most starred first

Data pipelines & ETL, page 9 of 13

populisengineagentsmanagementdb

Management CRUD in MongoDB

pragmastat

Pragmastat: Pragmatic Statistical Toolkit

pragzip

Parallel random access to gzip files

private-django-googledrive-storage

Storage implementation for Django that interacts with Google Drive

prj-shrd-object-storage

Add your description here

py-agent-ner

NER training-data generation built on py-agent-lib. Bring your own labels, LLM, and downstream trainer.

py-single-file-storage

Store some information within the current .py file for the distribution of a single-source-file Python application.

py-storage
pyAverager

Parallel Python Averager for Climate Data

pyBallMapper

Python implementation of the Ball Mapper algorithm.

pyFragility

Fragility function fitting for any data type, with misspecification-robust uncertainty.

pyJsonStorage

pyJsonStorage is a Python package designed to implement basic database functionalities using JSON (JavaScript Object Notation). This package provides various features such as creating a new database, inserting rows, updating, and deleting rows. It leverages the JSON module from Python's core libraries for managing JSON files.

pyLodStorage

python List of Dict (Table) Storage library

pyStorages

Simple data storages package

pybeeswarm

Beeswarm plots for python

pydantic-aioredis

Use your pydantic models as an ORM, storing data in Redis.

pydantic-airflow

Pydantic models for Apache Airflow

pydantic-airtable

A Python library for managing Airtable data using Pydantic objects

pydantic-storage

A lightweight, type-safe storage system for Pydantic models with support for multiple backends and persistent storage.

pydatastorage

A python data storage backend library.

pyddragon

Python wrapper library providing LoL ddragon data asynchronously

pyfstorage

File Storage

pygvcf2coverage

A python tool to extract coverage from gvcf files.

pylocalstorage

A package to store data on the hard disk (HD) and make it available to all Python applications running locally!

pyopenair

A simple python package to convert geo data to OpenAir format

python-ragic

Python Ragic API client for data loading and manipulation.

python-slimta-cloudstorage

Queue storage backend interfacing with cloud providers.

python-slimta-diskstorage

Adds a filesystem-based queue extension to python-slimta.

python-slimta-redisstorage

Adds a redis-based queue extension to python-slimta.

pywe-storage

Wechat Storage Module for Python.

pyworks-storage

Provide a variety of storage backends in a single library.

q1-storage

Simple local file storage backed by SQLite with optional AES-GCM encryption

qai-storage

Qai Storage API

qiniu-storage

Django storage power by qiniu store

quantumcore.storages

A collection of storage modules like for file management, metadata and more

radicale-storage-by-index

A regular radicale storage with an additional sqlite index on specified fields.

radicale-storage-decsync

DecSync storage plugin for Radicale

radicale-storage-etesync

DEPRECATED: merged into etesync-dav

radicale-storage-family-scheduler

Basic scheduling for users on the same server

radicale-timerange-indexed-storage

A time range indexed radicale storage

rag-evaluation-dataset

Generate QA pairs from JSON documents to evaluate RAG pipelines

raga-cli

Git for data scientists - manage your code and data together

raggen

RAG dataset generator

ragnar

lightweight Extract-Transform-Load (ETL) framework for Python 3+

raptus.mercurialstorage

provides an archetypes storage which stores files on the filesystem into a mercurial repository

rcs-storage

Client for the Research Data Storage Registry, Research Computing Services, University of Melbourne.

rdpurchasestorage

reshape data type in py language

rdspy

A small example package

rdvt-s3storage

Reserved package to prevent dependency confusion.

readonlystorage

A readonly wrapper around a ZODB storage

recc-storage

Recc Object Storage

redsolutioncms.django-hex-storage

Django hex storage is add-on on the top of Django file system. It has improvements that cover: international filenames (processed with pinyin module) and the same file names (django adds underscore to same filename, we add HEX hash) - hence the name :)

reids-json-storage

Redis json stroage.

relstorage_packer

Packs a History Free PostgreSQL RelStorage for ZODB.

reswarm

Aids users in publishing data to a Record Evolution Datapod

rimo-storage

好!

rl-test-storage-handlers

Storage handlers

roo-data-storage
roughrider.storage

File storage utility implementation

rusty-haystack

Fast Python bindings for Project Haystack — types, codecs (Zinc/Trio/JSON), filters, ontology, and Xeto support powered by Rust

s3-object-storage-client

s3 using tool

s3-storage-stats

S3 Storage Stats

s3-synapse-storage-provider

A storage provider which can fetch and store media in Amazon S3.

safe-s3-storage

S3 safe storage

sagru-storage

A simple Storage backend for Django that makes use of Amazon's S3 service.

sas-storage

A comprehensive Python helper for Azure Storage Blob and Queue operations with file explorer-like functionality

sayou-llm

LLM components for the Sayou Data Platform

scatac_fragment_tools

Tools for working with scATAC-seq fragment files.

segmentation-agent

AI-powered dataset segmentation agent for ML workflows

selectel-storage
senaite.storage

Sample storage add-on for SENAITE

sequana-coverage

Parallelise version of sequana_coverage standalone application.

simple-module-file-storage

Pluggable file upload + storage (local or S3 via extras) module for simple_module apps

simple-storage

File storage made easy

simple-storage-python-module

A small, easy to use module that helps you store data easily.

simple-storage-service

Assist in the handling of files on storages from different providers.

simpleasyncfilestorage
simplestorage

Storage client for local and gcloud

singlefile-storage

SingleFile Rest Storage for python.

skfeaturellm

LLM-powered feature engineering with scikit-learn API compatibility

smb-storage
snakemake-interface-storage-plugins

This package provides a stable interface for interactions between Snakemake and its storage plugins.

snakemake-storage-plugin-apd

A Snakemake storage plugin for the LHCb apd API.

snakemake-storage-plugin-azure

A Snakemake storage plugin to read and write from Azure Blob Storage

snakemake-storage-plugin-cached-http

Snakemake storage plugin for downloading files via HTTP with caching and rate limiting

snakemake-storage-plugin-deeporigin

A Snakemake storage plugin for S3 API storage (AWS S3, MinIO, etc.)

snakemake-storage-plugin-dirac

A Snakemake storage plugin for the DIRAC Data Management System (DMS).

snakemake-storage-plugin-fs

A Snakemake storage plugin that reads and writes from a locally mounted filesystem using rsync

snakemake-storage-plugin-ftp

A Snakemake plugin for handling input and output via FTP

snakemake-storage-plugin-gcs

A Snakemake storage plugin for Google Cloud Storage

snakemake-storage-plugin-git

A Snakemake storage plugin for downloading remote Git repositories.

snakemake-storage-plugin-http

Snakemake storage plugin for downloading input files from HTTP(s).

snakemake-storage-plugin-irods

A Snakemake plugin for handling input and output on iRODS

snakemake-storage-plugin-latch
snakemake-storage-plugin-lfs

Snakemake storage plugin for downloading Git LFS files with optional local caching

snakemake-storage-plugin-nersc

Snakemake plugin to interact with the NERSC filesystem

snakemake-storage-plugin-orcestra
snakemake-storage-plugin-pelican

A Snakemake Storage Plugin for Pelican Federations

snakemake-storage-plugin-rucio

A Snakemake storage plugin that handles files available through Rucio.

snakemake-storage-plugin-s3

A Snakemake storage plugin for S3 API storage (AWS S3, MinIO, etc.)

More in Data, search & knowledge