Data, search & knowledge

Data pipelines & ETL

1,206 agents · most starred first

Data pipelines & ETL, page 5 of 13

arcane-bigquery-storage

A package to use the BigQuery Storage Write API

arcane-storage
archetypes.ldapstorage

LDAP Storage for Archetypes

archetypes.memberdatastorage

A facade storage layer for Archetypes delegating to the Plone memberdata tool

asset-model-data-storage

Data storage library for asset model apps

atari-monk-keyval-storage

A simple JSON-based key-value storage library

atomstorage

API for storing and retrieving Atom entries.

atoti-client-storage-aws

Client part of the plugin to load CSV and Parquet files from AWS S3 into Atoti tables

atoti-client-storage-azure

Client part of the plugin to load CSV and Parquet files from Azure Blob Storage into Atoti tables

atoti-client-storage-gcp

Client part of the plugin to load CSV and parquet files from Google Cloud Storage into Atoti tables

atoti-server-storage-aws

Server part of the plugin to load CSV and Parquet files from AWS S3 into Atoti tables

atoti-server-storage-azure

Server part of the plugin to load CSV and Parquet files from Azure Blob Storage into Atoti tables

atoti-server-storage-gcp

Server part of the plugin to load CSV and parquet files from Google Cloud Storage into Atoti tables

atreal.filestorage.blobfile

Defines a common API to handle file objects independently of storage - Blobfile plugin

atreal.filestorage.common

Defines a common API to handle file objects independently of storage

auto-backup-storage

A Python script for backing up and synchronizing data between SSD and HDD with CPU usage monitoring.

autogenix-ETL-automation

A package for automating ETL processes

autonomous-dq-agent

A cloud-ready, ML-powered, rule-driven data quality agent.

average

Moving averaging schemes (exponentially weighted, polynomial-decay).

average-calculator

A library to calculate the average of marks

average-decision-ordering

Calculates the average decision ordering metric

average-n-numbers-calvine

A sample Averaging project

averagee-calculatorr

A library to calculate the average of marks

averagepackage-sanwariyia

To find avg

averager

A simple Python library to calculate averages of values.

aws-resource-validator-kinesis-video-webrtc-storage

Pydantic v2 models for AWS kinesis_video_webrtc_storage, shipped as a PEP 420 namespace extension of aws-resource-validator.

azure-storage

Microsoft Azure Storage SDK for Python

azure-storage-blob

Microsoft Azure Blob Storage Client Library for Python

azure-storage-blob-changefeed

Microsoft Azure Storage Blob ChangeFeed Client Library for Python

azure-storage-common

Microsoft Azure Storage Common Client Library for Python

azure-storage-file

Microsoft Azure Storage File Client Library for Python

azure-storage-file-datalake

Microsoft Azure File DataLake Storage Client Library for Python

azure-storage-file-share

Microsoft Corporation Azure Storage File Share Client Library for Python

azure-storage-helper

Azure Storage Helper

azure-storage-nspkg

Microsoft Azure Storage Namespace Package [Internal]

azure-storage-queue

Microsoft Corporation Azure Storage Queue Client Library for Python

azure-storage-utils

A utility for tracking volume in and out of azure storage containers

azurestoragewrap

A Python Wrapper for modeling client side encrypted azure storage tables, queues and blobs

b2-storage

Backblaze b2

backend-storage

提供了基于 Node.js 开发的 RESTful 接口,可以方便地进行数据的增删改查(CRUD)操作以及高效的数据搜索

batch-table-storage

Async batch functionality for Azure Table Storage

bayesian-average

Bayesian weighted averages for inconsistent data sets

bellmanford

Small extensions of the Bellman-Ford routines in NetworkX, primarily for convenience (https://networkx.github.io).

betterbeeswarm

Custom implementation of seaborn Beeswarm class

bgf-agents

Autonomous Data Factory Agents - Health Monitoring, Self-Healing, and Data Governance

boto3-dataclass-bedrock-data-automation

Dataclass for boto3 bedrock_data_automation.

boto3-dataclass-bedrock-data-automation-runtime

Dataclass for boto3 bedrock_data_automation_runtime.

bp-storage

Helper Library for various Dataset formats

bunnycdn-storage

A python module for the BunnyCDN Storage API

bunnycdn-storage-handler

A Python client for BunnyCDN Storage API

categorical-gpt

Transformation of categorical data to numerical feature vectors with Large Language Models

cbi-storages

Base CBI objects for storages (RethinkDB, Redix, Clickhouse, etc)

cbstorage

The code is a Python package for managing file storage on either AWS S3 or a local machine. It provides methods for listing, searching, and storing files. The package includes two classes, Aws and Local, which implement the same set of methods for accessing and manipulating files in S3 and a local directory.

ccsdspy

IO Interface for Reading and Writing CCSDS Data in Python.

ccstorage

Local storage: CCStorage, and simple local file string read/write: CCIO

cdsswarm

Concurrent CDS API downloader with TUI and script mode

chaosplatform-relational-storage

A library for relational database access from the Chaos Platform

chatgpt-to-sqlite

Import ChatGPT conversations into a SQLite database

cipher.encryptingstorage

ZODB storage wrapper for encryption of database records

ckanext-cloudstorage

Cloud storage for CKAN

ckanext-cloudstorage_api

Extension to manage CKAN resources in any S3-like object storage.

ckanext-storage

CKAN API Extension for OFS Storage

classifierAgent

A Python package for performing classification on datasets.

claude-data-skills

A comprehensive collection of data science, analysis, and engineering skills and scripts.

claude-to-sqlite

Convert a Claude.ai export to SQLite

cleaning-agent

Intelligent data cleaning agent for automated data quality improvement

cllm-data-curation

A package to visualize tokenization of text using HTML

cmis_storage

cmis_storage is a storage engine for Django to integrate CMIS-compatible services, like Alfresco

collectd-ceph-storage

Yet another Ceph Collectd Plugin in Python.

collective.atcassandrastorage

A AT field storage which stores values to a cassandra database

collective.googlesystemstorage

GSS saves the document types supported by the Google Docs service on the Google servers

collective.psc.blobstorage

Blob storage for PloneSoftwareCenter

collective.psc.externalstorage

External storage for PloneSoftwareCenter

collective.recipe.filestorage

This recipe aids the creation and management of multiple Zope 2 filestorages.

common-storage

An util for python backend storage packaging.

commonhealth-cloud-storage-client

Partner Client for CommonHealth Cloud Storage Service

commonhealth-storage-client

TODO

community-cloud-storage

Distributed storage for community archives using ipfs-cluster, Tailscale and Docker

coopstorage

Package for embedded storage systems

couchdb-storage

Use couchdb to store storage meta data

coverage2sql

Command to Read a coverage file and put the data in a SQL database

cpforager

Python package to handle data gathered from biologgers (GPS, TDR, AXY) attached to central-place foraging seabirds. The idea is to make movement ecology data a bit easier to process.

crams-storage

CRAMS API opensource crams storage package.

create-tables-storage

Libreria de creacion de tablas para un proyecto de migracion

cs-storage

A small package that is used by Compute Studio to read and write model results to google cloud storage.

cushy-storage

A data local persistence ORM framework.

dasscostorageclient

DaSSCo Storage API SDK

data-agent

Data Agent is a python library for pipelining real-time and historical data to and from industrial historians and control systems

data-agent-aspen-ip21

Aspen IP21 historian plugin for `data-agent` package

data-agent-osisoft-pi

Osisoft PI historian plugin for `data-agent` package

data-agent-zip

ZIP files plugin for `data-agent` package

data-designer-cache-llm-column

Data Designer plugin adding cache-enabled LLM text and structured column generators

data-enrichment-agent

Intelligent data enrichment agent for automated feature engineering

data-forager

Enabling random access to large datasets on disk for PyTorch training and other use cases

data-juicer-agents

A multi-agent data processing system built on AgentScope and Data-Juicer

data-science-llm-core
data4llm

Add apply_multi and other bugs fix

datacatalog-object-storage-processor

A package for performing Data Catalog operations on object storage solutions

datacatalog-storage

Catalog-based data storage system with pluggable serializers

dataframes-haystack

Haystack custom components for your favourite dataframe library.

More in Data, search & knowledge