GigaToken: ~1000x faster Language model tokenization

Gigatoken claims GB/s tokenization by moving the hot path into optimized Rust with SIMD, caching, and less Python overhead.

The project presents Gigatoken as a drop-in replacement for HuggingFace Tokenizers and tiktoken, with a faster native API when it can read files directly. Its benchmarks show the biggest gains on common BPE tokenizers, including GPT-2, Qwen, Llama, DeepSeek, Phi, and OLMo families. Compatibility mode is described as exact-output focused but slower than the full Gigatoken API. The author lists limits: WordPiece is unsupported, SentencePiece is less optimized, Windows is lightly tested, and CJK-heavy data is slower. HN · GitHub AI's note