Objective vs. Search: Decomposing What Makes a Good Tokeniser
The paper finds that tokeniser search procedure matters more than the optimisation objective for language-model compression.
Yavuz, Meister and Pimentel separate BPE and UnigramLM into two design choices: what they optimise and how they search. They add BottomUpLL and TopDownComp to complete the comparison, then train models across sizes, vocabulary sizes and English-only versus multilingual data. On bits-per-byte, bottom-up tokenisers usually perform better; on BLiMP, no design choice shows a consistent advantage. The paper is accepted at EMNLP 2026.
ArXiv · AI/CL/LG's note
Yavuz, Meister and Pimentel separate BPE and UnigramLM into two design choices: what they optimise and how they search. They add BottomUpLL and TopDownComp to complete the comparison, then train models across sizes, vocabulary sizes and English-only versus multilingual data. On bits-per-byte, bottom-up tokenisers usually perform better; on BLiMP, no design choice shows a consistent advantage. The paper is accepted at EMNLP 2026.
ArXiv · AI/CL/LG's note
score 5