Data, search & knowledge

Data pipelines & ETL

1,206 agents · most starred first

Data pipelines & ETL, page 7 of 13

gcp-storage-emulator

A stub emulator for the Google Cloud Storage API

gdpstorage

Storage Engine for Django Project based on Google Drive Personal Drive

geminisqlite

A database framework for exploring genetic variation

genro-storage

Unified storage abstraction for Genropy framework

ghg.object-storage

Dummy package for GHGSat

ghg.storage

Dummy package for GHGSat

github-storage

A Python package that uses GitHub repositories as a database for storing and retrieving data.

github-storage-system

To use github to store files

gitstorage

A library that uses git version control system to store and synchronize files

global-storage-json-dict
gollama

Django implementation for golink

good-object-storage

Basic S3 client with some helpful utilities

google-apitools-storage-v1

Autogenerated apitools library for storage

google-cloud-bigquery-storage

Google Cloud Bigquery Storage API client library

google-cloud-storage

Google Cloud Storage API client library

google-cloud-storage-control

Google Cloud Storage Control API client library

google-cloud-storage-transfer

Google Cloud Storage Transfer API client library

google-cloud-storagebatchoperations

Google Cloud Storagebatchoperations API client library

google-cloud-storageinsights

Google Cloud Storageinsights API client library

gottwall-storage-redis

Redis storage for GottWall statistics aggregator

gpt-data-dictionary

data-dictionary is a Python package that automates the creation of data dictionaries utilizing LLM. It generates human-readable variable names and provides comprehensive metadata, making it an essential tool for data management, enhancing data interpretability and quality assurance.

gpt-scientist

A library for processing textual datasets with large language models.

gptables

Simplifying good practice in statistical tables.

gptc

General-purpose text classifier

granite-storage

Storage abstraction for SQLAlchemy models with local and S3 backends, streaming uploads, and orphan garbage collection.

gtrendspy

Private package for pulling and merging Google Trends data

guillotina-dynamictablestorage

Dynamic storages based on tables instead of databases

guillotina-gcloudstorage

guillotina gcloud storage support

guillotina-localstorage

Local FS storage support for Guillotina

guillotina-s3storage

s3 guillotina storage support

gumo-storage

Gumo Storage Library

h2ogpte-migration

Automated tool for migrating h2oGPTe collections

haagentus

An statiscian package for python & very easy version

hashstorage

Implementation of Simple-HashStorage.

haste-storage-client

Client for the HASTE storage plaform: http://haste.research.it.uu.se

hawkapi-storage

File storage for HawkAPI — local, S3, GCS, Azure backends, pre-signed URLs, streaming uploads

hcs-storage

a storage class wrapping a dict.

heaserver-storage

a service manages storage information details

hf-storage

Hugging Face dataset-backed cloud file storage library

hierarchical-storage

package for tree like structure sorted by category with named data

hmd-entity-storage

basic storage library for entities and entity definitions

hopeit.azure.blob-storage

Hopeit Engine Azure Blob Storage Toolkit

hopeit.fs-storage

Hopeit Engine File System Storage Toolkit

hopeit.redis-storage

Hopeit Engine Redis Storage Toolkit

huscy.project-storage
idspy

Python library providing shared code for IDS datasets

idspy-dictionaries

IMAS Data Dictionaries converted to Python Dataclass used in the IMAS-GK/IDSPy parser

idspy-toolkit

Tools to manipulate the IDSPy Dictionaries

imap-storage

Use your Email-account as Storage for data structures and files

incstorage

Incremental Storage

incuna-storages

A collection of django storage backends.

indico-plugin-storage-s3

S3 storage backend for Indico

indico-plugin-storage-xrootd

XRootD/EOS storage backend for Indico

infi.storagemodel

A high-level library for traversing the OS storage model.

ingest4llm

Ingest different data sources into structured data for LLM

inoagent-registry

Функционал получения актуального списка иноагентов с сайта МинЮста РФ

invenio-files-multisum-storage

PyFS File Storage For Invenio with support for multiple checksum algos

isotoma.zope.groupreadableblobstorage

Make blob storage group readable

iterage

Yet another iterator utils lib

iwm.recipe.blobstorage

ZC Buildout recipe for defining a blob-storage

jf-agent

An agent for collecting data for jellyfish

json-storage-manager

JSON Storage Manager for JSON data stored in text-files

jsonstorage

JSON Storage is a simple lightweight data storage using JSON format in Python.

jupyter_erp5_storage

ERP5 Storage for jupyter

just-storage

Python SDK for JustStorage object storage service

kbcstorage

Client for Keboola Storage API

key-multivalue-storage

A simple library that allows the JSON storage of one key to multiple values.

kili-common-storage

A storage package for kilimall lite

koverage

Quickly get coverage statistics given reads and an assembly

langchain-airbyte

An integration package connecting Airbyte and LangChain

langchain-etl
langchain-iceberg

LangChain integration for Apache Iceberg with native PyIceberg API support

langchain-lakefs

An integration package connecting LakeFS and LangChain

langchain-onelake

LangChain toolkit for Microsoft Fabric OneLake — workspace, file, table, directory, and item operations powered by the Fabric MCP Server.

lazymappingstorage

A dict like storage backed by a yaml file or similar method

ldc-openai

Python3 library containing plugins for llm-dataset-converter specific to OpenAI.

leancloud-better-storage

Better ORM wrapper of leancloud storage python sdk.

lineage-suggestion-agent

Add your description here

lineagentic-flow

Lineagentic-flow is agentic ai approach for building data lineage across diverse data processing scripts including python, sql, java, airflow, spark, etc.

litestar-storages

Async file storage abstraction library with optional Litestar integration

lithaxor-bullmq-python

BullMQ for Python (LitHaxor fork) — adds JobScheduler, global rate limit/concurrency, per-job cancellation, and QueueEvents streams.

livingbio-django-hashedfilenamestorage

A Django storage backend that names files by hash value.

livingbio-django-oss-storage

Django Aliyun OSS (Object Storage Service) storage

llama-backup

Backup and restore utilities for LlamaAI data and configurations.

llama-cleanup

A package to process addresses and filter out noise

llama-cli

Llama CLI fetches and preprocesses learning data

llama-index-ingestion-ray

llama-index Ray ingestion pipeline

llama-index-packs-diff-private-simple-dataset

llama-index packs diff private simple dataset

llama-index-packs-llama-dataset-metadata

llama-index packs llama_dataset_metadata integration

llama-index-packs-raft-dataset

llama-index packs RAFT Dataset paper implementation

llama-index-readers-airbyte-cdk

llama-index readers airbyte_cdk integration

llama-index-readers-iceberg

llama-index readers iceberg integration

llama-index-readers-pathway

llama-index readers pathway integration

llama-index-readers-structured-data

llama-index readers structured_data integration

llama-mlx-pipeline

Data processing pipeline using MLX (scraper, chunker, extractor).

llama-mlx-pipeline-llamasearch

Data processing pipeline using MLX (scraper, chunker, extractor).

llamadb-llamasearch

Next-Gen Hybrid Python/Rust Data Platform with MLX

llamass

LLAMASS is a Loader for the AMASS dataset

llm-annotator

An easy-to-extend LLM annotator for robust, resumable data annotation.

llm-batch-annotate

Batch-oriented LLM annotation workflows for tabular datasets with OpenAI Batch support.

More in Data, search & knowledge