Hugging Face rebuilds its tokenizer from the bytes up

Hugging Face has published a release candidate of its tokenizers library that encodes text between three and 30 times faster than the version most projects are running, while producing exactly the same output. The engineering post, published on 21 September, measures v1 against v0.23 on an Apple M4 Max across ten model families, eight of which use byte pair encoding.

What actually got faster

Most of the win comes from throwing away a regex engine. Byte pair encoding splits text into pre-tokens using a regular expression that ships with the model and never changes at runtime, so there is no reason to interpret it on every call. v1 replaces it with a hand-written splitter called bitcannon that treats the input as parallel streams of bits and finds boundaries with SIMD instructions, deciding 64 bytes per register operation, the same idea that made its name in JSON parsing. Hugging Face says the change covers the common byte-level grammars, including those behind GPT-2, DeepSeek and the cl100k and o200k families, and that a tokenizer whose pattern is not among them keeps the regex path and none of the speed-up.

The rest comes from not repeating work. A thread-local cache maps each pre-token to its finished token IDs, so a word already seen skips the merge process entirely. The merge loop now runs out of a scratch buffer owned by the caller instead of allocating for every call, links the symbols it is merging by position inside one flat buffer rather than moving data, and packs each candidate pair into a single 64-bit integer so choosing the next merge is an integer comparison rather than a branch. Pre-tokens are also handled in batches, one model call per batch instead of one per pre-token.

Parallelism was rebuilt with them. One shared tokenizer can encode from many threads at once, with each thread drawing its own scratch buffer and its own cache from a sub-pool, which removes the single lock that used to line them up. Hugging Face reports that v1 scales at 76 per cent of linear across eight workers. The crate itself was also split: one library became a workspace where only the encoding runtime is required and the serialisation, conversion and training parts are linked when an application actually needs them.

How the numbers were taken

The figures come from tokbench, a benchmark harness the team built for this work and published alongside it. Every engine runs the same timing loop, vocabulary loading is timed separately so it never lands inside an encode measurement, the output IDs are hashed and checked against the baseline exactly, and workers are pinned to eight distinct physical cores rather than sibling threads. The post also flags a trap in this kind of test: repeatedly encoding one document measures a fully cached path, while encoding a stream of distinct documents measures something harder, and both are often described as warm. The headline figures use distinct documents.

One caveat sits on the measurement side. The release candidate is a Rust crate, and the Python bindings that most projects will actually call wrap the same code but add per-call overhead that none of these timings include.

What is still missing

The release candidate is on crates.io, so installing it is a one-line change and the API is unchanged, so the only thing that moves is which build you get. The rest of the way to 1.0.0 is not finished. Hugging Face still wants a single encoding implementation shared by training and inference so the two cannot disagree, offsets and masks computed only when they are requested, reworked normalisers, simpler Python bindings that keep working under free-threaded CPython, and inference-only C and C++ bindings for ExecuTorch and llama.cpp, with JVM, Swift and Go bindings possible after that. Past 1.0.0 the team is prototyping tok-devices, which would move encoding and batch decoding onto the GPU; it calls that exploratory.

The post credits IBM, NVIDIA and the ExecuTorch team for patches and for testing across a wide range of hardware. It also names the smaller tokenizer projects whose ideas fed the rewrite, among them gigatoken, tiktoken, kitoken, tokie, fastokens, wordchipper and ai-tokenizer. Hugging Face’s own summary is that tokenizers was "nowhere near the performance it could have had", and that it intends the library to be one worth contributing to.

Our opinion

The number worth caring about here is not the 30. It is that v1 emits exactly the same token IDs as v0.23. A performance rewrite that keeps the output bit-identical asks nothing from the ecosystem in return: no retraining, no re-tuning, no re-running of evaluation suites to explain a shifted score, no migration guide. That is rare, and it is why this matters more than the speed. Libraries that get dramatically faster usually charge for it somewhere else, in changed semantics, a new dependency or a broken plugin path. Hugging Face spent the effort keeping the contract, then published the credit list naming the smaller projects whose work it borrowed. Both choices are worth more to the field than another benchmark chart.

Read the range honestly and it is a spread, not a promise. Three times on t5-base and 30 times on the gpt2 family is not measurement noise, it is the difference between a model whose splitter grammar made it into the hand-written fast path and one that still runs the regex. If your pipeline uses a model outside that set you get none of this, and a headline will not tell you so. Teams feeding enormous training sets or serving long prompts should time their own model on their own hardware before assuming anything. Keep the other half of the framing in mind too: for most workloads the tokenizer was never the bottleneck. This fixes the intake pipe, and it only shows up where the pipe was actually holding the flow back.