Data, search & knowledge

Data pipelines & ETL

1,206 agents · most starred first

Data pipelines & ETL, page 8 of 13

llm-batch-processor

A package to process CSV text data in batches using OpenAI API

llm-batch-py

Cron-friendly batch LLM processing for Polars.

llm-compact-serializer

Dynamic serializer for LLM Compact Hierarchical Formats.

llm-datagen

极简高性能流式数据加工库

llm-dataset-converter

Python3 library for converting between various LLM dataset formats.

llm-dataset-converter-all

Meta-library that combines all llm-dataset-converter libraries.

llm-explorer

A Lakehouse LLM Explorer. Wrapper for spark, databricks and langchain processes

llm-feat

Automated feature engineering using Large Language Models (LLMs) for tabular data

llm-feature-gen

Generate interpretable feature schemas and tabular exports from text, images, tabular data, and video with LLMs

llm-hygiene

a data preprocessing toolkit that makes it easy to create common LLM-related data structures; from training data to chain payloads!

llm-join

Fuzzy join pandas DataFrames using LLM scoring and embedding retrieval — record linkage, entity resolution, approximate join

llm-jsonl-converter

A Python package for converting unstructured text into JSONL format using LLMs.

llm-labeling-ui

LLM Labeling UI is an open source project for large language model data labeling

llm-to-corpus

Large language model to corpus

llm-web-crawler

LLM data collection and synthetic fine-tuning dataset pipeline

llm2dict

Enable large language models to output structured data.

llm2json

Web Based Multi Purpose Annotation Software

llmabba

LLM-ABBA: mining time series via symbolic approximation and large language models

llmaug

🪄 Dataset augmentation using LLMs

llmtag

Use LLMs to label any textual dataset

llmtopic

A library for extracting topics from text documents using llms.

lndb_storage

Storage → object.

localStoragePro

A familiar API from the Web, adapted to storing data locally with Python.

localStoragePy

A familiar API from the Web, adapted to storing data locally with Python.

localstorage-flask

A Flask extension for local storage

log-storage

Log handler to store messages with a database record and optional file storage.

logyca-azure-storage-blob

An integration package created by the company LOGYCA to interact with the blob container files in your Azure Storage account.

lsm-storage-engine-key-value-store

A persistent key-value storage engine based on LSM-tree architecture.

magento-dump

Magento Database Code end Media Dump & Table Analyzer Tool

mapping-agent

LLM-driven agent for creating detailed column-to-attribute mappings

mara-storage

Configuration of storage connections for mara

maverage

Light implementations for using moving averages in real time.

mdspy

Mnubo Data Science Library

minio-storage

minio simplified

mirage-database-service

A brief description of your package

mlgpt

A Causal Experiment Suite

mllmdata

Easily turn a set of image urls to an image dataset

mmirage

Modular Multimodal Intelligent Reformatting and Augmentation Generation Engine - Advanced platform for processing datasets using generative models including vision-language models.

mogilefs_storage

MogileFS FileStorage for Django

mosaik.HDF5-Storage

Store any simulator metric into a HDF5 database using a common scheme.

movingaverage

Provides a moving average over a data set.

msaStorageDict

Dict with a Storage Backend like redis or Zookeeper

msaas-file-storage

Dual-stack file storage SDK wrapping AWS S3

multi-storage-client

Unified high-performance Python client for object and file stores.

multicloud-storage

统一操作 MinIO/OSS/S3 兼容存储的工具包

munch-storage-swift

Swift storage backend for munch-core.

mypy-boto3-storagegateway

Type annotations for boto3 StorageGateway 1.44.0 service generated with mypy-boto3-builder 8.12.0

nameko-objectstorage

A dependency for IBM Bluemix Object Storage

nanoBragg-noise

A command-line tool to add noise to simulated H5 data.

nbforager

Python package designed to assist in working with the Netbox REST API. The filter parameters are identical to those in the Web UI filter form. It replaces brief data with full information, and Netbox objects are represented as a recursive multidimensional dictionary.

ndn-storage

An NDN Storage Library for Applications in Python 3

netsight.cloudstorage

Allows you to store large files in the cloud

network-spatial-coherence

Network Validation using the Spatial Coherence Framework.

ngsfragments

Python package for Next Generation Sequencing fragment manipulation

nonestorage

Simple library that provides local storage folder detect

nosql-storage-wrapper

A simple library for easy work with key-value storages and document databases

notebookManagerPackageNT

A package to manage the execution of multiple notebooks

oarepo-enrollment-permissions

OArepo Enrollment library for record metadata validation

oarepo-enrollments

OArepo Enrollments library for record metadata validation

object-storage

Python library for accessing files over various file transfer protocols.

object-storage-charmlib

Common charm lib for providers and requirers of object storage relation interfaces

object-storage-client

Unified object storage client API

object-storage-proxy

<object-storage-proxy ⚡> Yet Another Object Storage Proxy

octopus-cloud-storage-hub

Octopus Cloud Storage Hub

oem-storage-codernitydb

OpenEntityMap - CodernityDB Storage

oem-storage-file

OpenEntityMap - File Storage

ohmysportsfeedspy

A Python wrapper for the MySportsFeeds Sports Data API

omni-storage

A unified Python interface for file storage.

openai-gabriel

LLM-based library to measure quantitative attributes on qualitative data

openai-to-sqlite

Save OpenAI API results to a SQLite database

openai-to-sqlite-api2d

Save OpenAI API results to a SQLite database

openair

This project is to fetch data from the official Silverlight application of Ministry of Environmental Protection of China (http://106.37.208.233:20035/), which publish realtime air quality. 本项目旨在方便地获取官方发布的中国空气质量数据。

openair-rs-py

Python bindings for OpenAir airspace file parser

openproximity-agent2

A data collector that pushes records into a centralized server.

optuna-firestore-storage

It helps optuna to use Firestore as a backend DB.

orbis-plugin-storage-cache-webservice-results

The storage plugin to cache webservice results.

orbis-plugin-storage-csv-result-list

The csv_result_list storage plugin for Orbis

orbis-plugin-storage-export-results

The storage plugin to export Orbis results into different formats.

orbis-plugin-storage-html-pages

The html_pages storage plugin for Orbis

orbis-plugin-storage-single-view

The single_view storage plugin for Orbis

orbit-storage
our-storage

Erasure-coded storage backends with Merkle integrity verification for the ourochronos ecosystem

outcome-pypicloud-storage-gcs

A fork-safe verion of the GCS Storage backend for PyPICloud.

p-toon-llm

Token-oriented object notation (TOON) for compact, reversible JSON encoding

palmers-agents-general

A package to preprocess data for Palmer's project

pandas-storage-compressor

Package for compress storage dataframe

pandora-storage-server-client

Pandora Storage Server Client.

papai-unified-storage

Supercharged fsspec filesystem classes with higher-level functions and utilitarians.

paperstorage

A module to create paper backups for arbitrary data that are recoverable by simple means

parag-forest

A small Random Forest Regressor implemented in pure Python and numpy

paragon-engine

High-performance parallel graph processing engine (C++ + Python)

paragres

Utility for synchronizing parallel PostgreSQL databases on Heroku, local, and remote servers

peewee-storages

Support for FileField and storage backends in Peewee

pepdbagent

A python-based database manager for portable encapsulated projects

pepefragment

Async Python client for your FraglyFragment API service

perceptra-storage

Unified storage adapter interface for multi-cloud storage backends

pfstorage

object storage interface

pgstoragelogger

Like Python logger, but partiotioned PostgreSQL tables are used for saving.

pilot-platform-object-storage

Python library for interacting with multiple object storage APIs.

podspy

A SciKit for process oriented data science

More in Data, search & knowledge