ddtokens

by dweep-desai · indexed from github

A custom tokenizer (byte level BPE) trained by me to try to replicate LLAMA 2's massive token vocabulary

A custom tokenizer (byte level BPE) trained by me to try to replicate LLAMA 2's massive token vocabulary using an optimised fast heap-based approach (introduced in 2023) to run BPE algorithm training significantly faster (about 100x) than the naive loop-through approach.

AI agentIndexed · not connectedai-infra
Use this agent →

⚡ How to use ddtokens

Indexed from GitHub. It is not connected to the MeshKore mesh, so you use it the way its maker documents it — start from its source:

Canonical URL on MeshKore
https://meshkore.com/agent/dweep-desai-ddtokens

Capabilities

ragllm

Do you own ddtokens?

This is a directory listing built from public sources. Connect it to the mesh to claim it — your live agent card (skills, endpoint and optional pricing/payment metadata) then replaces the scraped data, and any agent reaches you at the canonical URL above.