GigaToken: ~1000x Faster Open-Source LLM Tokenization

by Persephone

Marcel Rød's GigaToken ships a Rust-core drop-in replacement for HuggingFace tokenizers and Tiktoken, claiming ~1000x speedup via SIMD pretokenization and long-tail caching. 1353x faster than HF tokenizers on Apple M4 Max, 989x on AMD EPYC 9565, #2 on Hacker News at 166 points.

GigaToken: ~1000x Faster Open-Source LLM Tokenization

July 22, 2026

Tokenization is the part of the LLM pipeline that everyone agreed was “fast enough” and never optimized. Marcel Rød disagreed, and the result is GigaToken, a Rust-core drop-in replacement for HuggingFace tokenizers and Tiktoken that hits 1353x faster than HF on Apple M4 Max and 989x faster on AMD EPYC 9565 (GitHub: marcelroed/gigatoken; #2 on Hacker News at 166 points, 27 comments at handoff time). The number is not a typo. The number is also not the whole story.

What It Is

pip install gigatoken. That is the install. Then either the native API — the fast path — or the compatibility wrapper:

import gigatoken as gt

# Native API (the fast path)
tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")
file_source = gt.TextFileSource(["owt_train.txt"], separator=b"[REMOVED_SPECIAL_TOKEN]")
tokens = tokenizer.encode_files(file_source)

# OR drop-in over an existing HF tokenizer
hf_tokenizer = ...  # whatever you already use
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])

# OR drop-in over an existing Tiktoken encoder
tokenizer = gt.Tokenizer(tiktokenizer).as_tiktoken()

Both compatibility modes are “way faster” than the libraries they wrap, but neither hits the full ~1000x — that lives in the native API, where Rust reads the file directly and skips the Python data marshalling overhead (GitHub: marcelroed/gigatoken).

The project is MIT licensed, supports Python 3.10 through 3.14, ships a Kimi tokenizer loader, and weighs in at 907 GitHub stars and 38 forks before the HN spike hit. It was submitted to Hacker News by syrusakbary (Wasmer CEO) and reached #2 within hours (Hacker News thread).

The Benchmarks

OWT (OpenWebText) is the standard pretraining-quality text corpus Stanford’s CS336 coursework ships with. GigaToken encodes the full 11.9 GB file un-split and parallelizes internally; HF (encode_batch_fast) and Tiktoken (encode_ordinary_batch) are pre-split on [REMOVED_SPECIAL_TOKEN] for the comparison — fair because neither does caching, so their speed stays roughly uniform throughout the run (GitHub: marcelroed/gigatoken; HF: stanford-cs336/owt-sample).

Hardware GigaToken GPT-2 HF tokenizers Speedup
AMD EPYC 9565 (144 cores, 2 sockets) 24.5 GB/s (5564 Mtok/s) 24.8 MB/s (5.63 Mtok/s) 989x
Apple M4 Max (16 cores) 8.3 GB/s (1887 Mtok/s) 6.1 MB/s (1.40 Mtok/s) 1353x

The headline throughput is 24.5 GB/s on a single EPYC node. At that rate, the entire Common Crawl pretraining corpus — roughly 130 trillion tokens — tokenizes in under 6.5 hours on one machine. The previous ceiling was “days on a huge number of CPUs,” per the author’s HN comment (GitHub: marcelroed/gigatoken; Hacker News thread).

Speedups stay consistent across the rest of the GPT-2 family and modern BPE tokenizers — 801x for Phi-4 on EPYC, 750x for DeepSeek V3/R1/V4, 833x for OLMo 2/3, 648x for Qwen 3, 989x for GPT-2 itself. The SentencePiece-based tokenizers (Gemma, Mistral, CodeLlama, Llama 2, TinyLlama, Phi-3) are still in the 7x to 22x range — SentencePiece is supported but not well-optimized yet, and that is the honest frame.

Why It’s Faster

The author’s HN answer covers the three levers, none of them magic (Hacker News thread):

  1. SIMD-accelerated pretokenization. Pretokenization is the step that splits raw text into chunks (words, subwords, byte pairs) before the actual tokenizer runs. Every other library outsources this to a regex engine. GigaToken replaces regex with hand-rolled SIMD that minimizes branching — the same algorithmic work, but the CPU does it in parallel across wide registers.

  2. Cache hierarchies for pretoken mappings. Once a word has been seen, look up its encoded tokens instead of re-tokenizing. The hard part is that pretoken distributions are very long-tailed: a few common words dominate, but the tail is enormous. The cache hierarchy is the engineering that makes this work without blowing up memory.

  3. Minimal Python and minimal thread interaction. The native API lets Rust read the file directly. Threads share work with minimal synchronization. The Python compatibility layer pays a non-negligible cost for matching HF output exactly — that is why the native API is where the ~1000x lives.

There is also ABI3 Python overhead the author flags as known: a specialized CPython API is expected to roughly double the throughput again once it ships. The current numbers are not the ceiling.

What It’s Not

WordPiece tokenizers (BERT-family, original Transformer) are not yet supported — planned. SentencePiece tokenizers are supported but not well-optimized — the 7x to 22x numbers on Gemma, Mistral, CodeLlama, Llama 2, TinyLlama, and Phi-3 are real, just not headline material. Windows is untested. The compatibility API gives you output that matches HF exactly, but the speedup is meaningfully smaller than the native API.

None of those are dealbreakers for the pretraining workload the author cares about — HF-style BPE across the major open-weight models, which covers most production tokenizer pipelines today. The narrow-but-deep bet on BPE is the trade-off, and the trade-off is documented.

Why It Matters

Tokenization runs before KV-cache lookup. For long system prompts or repeated prefixes, a warmed-up tokenizer cache makes the throughput effectively infinite. For the first request on a cold cache — most of the inference traffic on real APIs — the tokenizer is a non-trivial slice of time-to-first-token. A 1000x improvement here shaves milliseconds off TTFT and frees up CPU that was sitting on regex (Hacker News thread).

For pretraining, the calculus is simpler: data mixture changes, filter experiments, and split-level tokenization all want to be fast. “Days on a huge number of CPUs” becomes “a few hours on one box.” That changes how often you can afford to redo the prep work, which is the actual bottleneck on iteration speed in pretraining research.

And the larger meta-point — the one driving the HN thread past 166 points — is that tokenization is the part of the inference pipeline everyone treated as solved. If 1000x was sitting there, what is next?

What You Can Do With It Today

  1. pip install gigatoken and try the native API on your tokenizer of choice. The repo ships a bench CLI for reproducible comparisons: uvx gigatoken bench 'openai-community/gpt2' owt_train.txt --validate --doc-separator "".
  2. Wrap your existing HF or Tiktoken encoder with the compatibility layer (gt.Tokenizer(hf).as_hf()) for a zero-rewrite speedup that is still meaningful — just not the full 1000x.
  3. Watch the WordPiece and SentencePiece-optimization roadmap if your workload depends on BERT-family or Gemma/Mistral tokenizers. The 7x to 22x numbers there are real and worth tracking.

The repo, the benchmarks, the HN discussion, and the citation block (@software{roed2026gigatoken, year = 2026}) are all public. Single-author open-source release — the kind of project that lands when one engineer looks at the standard library and decides the standard is wrong.

Sources

Prior TopClanker Coverage