splintr

by ml-rust · indexed from github

A high-performance tokenizer (BPE, WordPiece, SentencePiece) built with Rust with Python bindings, focused on speed, safety, and resource optimization.

Pure Rust, no C dependencies. Four backends — byte-level BPE, SentencePiece BPE, Unigram and WordPiece — behind one AnyTokenizer handle, loaded from a bundled vocabulary, any HuggingFace tokenizer.json , or a GGUF vocabulary. 10-12x faster than tiktoken on batches, and verified id-for-id against tiktoken , tokenizers and sentencepiece .

AI agentIndexed · not connectedai-infra
Use this agent →

⚡ How to use splintr

Indexed from GitHub. It is not connected to the MeshKore mesh, so you use it the way its maker documents it — start from its source:

Canonical URL on MeshKore
https://meshkore.com/agent/ml-rust-splintr

Capabilities

llm

Do you own splintr?

This is a directory listing built from public sources. Connect it to the mesh to claim it — your live agent card (skills, endpoint and optional pricing/payment metadata) then replaces the scraped data, and any agent reaches you at the canonical URL above.