Data pipelines & ETL
1,206 agents · most starred first
Data pipelines & ETL, page 8 of 13
A package to process CSV text data in batches using OpenAI API
Cron-friendly batch LLM processing for Polars.
Dynamic serializer for LLM Compact Hierarchical Formats.
极简高性能流式数据加工库
Python3 library for converting between various LLM dataset formats.
Meta-library that combines all llm-dataset-converter libraries.
A Lakehouse LLM Explorer. Wrapper for spark, databricks and langchain processes
Automated feature engineering using Large Language Models (LLMs) for tabular data
Generate interpretable feature schemas and tabular exports from text, images, tabular data, and video with LLMs
a data preprocessing toolkit that makes it easy to create common LLM-related data structures; from training data to chain payloads!
Fuzzy join pandas DataFrames using LLM scoring and embedding retrieval — record linkage, entity resolution, approximate join
A Python package for converting unstructured text into JSONL format using LLMs.
LLM Labeling UI is an open source project for large language model data labeling
Large language model to corpus
LLM data collection and synthetic fine-tuning dataset pipeline
Enable large language models to output structured data.
Web Based Multi Purpose Annotation Software
LLM-ABBA: mining time series via symbolic approximation and large language models
🪄 Dataset augmentation using LLMs
Use LLMs to label any textual dataset
A library for extracting topics from text documents using llms.
Storage → object.
A familiar API from the Web, adapted to storing data locally with Python.
A familiar API from the Web, adapted to storing data locally with Python.
A Flask extension for local storage
Log handler to store messages with a database record and optional file storage.
An integration package created by the company LOGYCA to interact with the blob container files in your Azure Storage account.
A persistent key-value storage engine based on LSM-tree architecture.
Magento Database Code end Media Dump & Table Analyzer Tool
LLM-driven agent for creating detailed column-to-attribute mappings
Configuration of storage connections for mara
Light implementations for using moving averages in real time.
Mnubo Data Science Library
minio simplified
A brief description of your package
A Causal Experiment Suite
Easily turn a set of image urls to an image dataset
Modular Multimodal Intelligent Reformatting and Augmentation Generation Engine - Advanced platform for processing datasets using generative models including vision-language models.
MogileFS FileStorage for Django
Store any simulator metric into a HDF5 database using a common scheme.
Provides a moving average over a data set.
Dict with a Storage Backend like redis or Zookeeper
Dual-stack file storage SDK wrapping AWS S3
Unified high-performance Python client for object and file stores.
统一操作 MinIO/OSS/S3 兼容存储的工具包
Swift storage backend for munch-core.
Type annotations for boto3 StorageGateway 1.44.0 service generated with mypy-boto3-builder 8.12.0
A dependency for IBM Bluemix Object Storage
A command-line tool to add noise to simulated H5 data.
Python package designed to assist in working with the Netbox REST API. The filter parameters are identical to those in the Web UI filter form. It replaces brief data with full information, and Netbox objects are represented as a recursive multidimensional dictionary.
An NDN Storage Library for Applications in Python 3
Allows you to store large files in the cloud
Network Validation using the Spatial Coherence Framework.
Python package for Next Generation Sequencing fragment manipulation
Simple library that provides local storage folder detect
A simple library for easy work with key-value storages and document databases
A package to manage the execution of multiple notebooks
OArepo Enrollment library for record metadata validation
OArepo Enrollments library for record metadata validation
Python library for accessing files over various file transfer protocols.
Common charm lib for providers and requirers of object storage relation interfaces
Unified object storage client API
<object-storage-proxy ⚡> Yet Another Object Storage Proxy
Octopus Cloud Storage Hub
OpenEntityMap - CodernityDB Storage
OpenEntityMap - File Storage
A Python wrapper for the MySportsFeeds Sports Data API
A unified Python interface for file storage.
LLM-based library to measure quantitative attributes on qualitative data
Save OpenAI API results to a SQLite database
Save OpenAI API results to a SQLite database
This project is to fetch data from the official Silverlight application of Ministry of Environmental Protection of China (http://106.37.208.233:20035/), which publish realtime air quality. 本项目旨在方便地获取官方发布的中国空气质量数据。
Python bindings for OpenAir airspace file parser
A data collector that pushes records into a centralized server.
It helps optuna to use Firestore as a backend DB.
The storage plugin to cache webservice results.
The csv_result_list storage plugin for Orbis
The storage plugin to export Orbis results into different formats.
The html_pages storage plugin for Orbis
The single_view storage plugin for Orbis
Erasure-coded storage backends with Merkle integrity verification for the ourochronos ecosystem
A fork-safe verion of the GCS Storage backend for PyPICloud.
Token-oriented object notation (TOON) for compact, reversible JSON encoding
A package to preprocess data for Palmer's project
Package for compress storage dataframe
Pandora Storage Server Client.
Supercharged fsspec filesystem classes with higher-level functions and utilitarians.
A module to create paper backups for arbitrary data that are recoverable by simple means
A small Random Forest Regressor implemented in pure Python and numpy
High-performance parallel graph processing engine (C++ + Python)
Utility for synchronizing parallel PostgreSQL databases on Heroku, local, and remote servers
Support for FileField and storage backends in Peewee
A python-based database manager for portable encapsulated projects
Async Python client for your FraglyFragment API service
Unified storage adapter interface for multi-cloud storage backends
object storage interface
Like Python logger, but partiotioned PostgreSQL tables are used for saving.
Python library for interacting with multiple object storage APIs.
A SciKit for process oriented data science