Latent Core Tokenizer: Compress, but Meaningfully
LCT reports better multilingual tokenization quality without cutting languages unevenly.
The paper introduces a tokenizer that first finds reusable linguistic units, then builds the shared vocabulary. Tested across 104 languages with a 200K-token vocabulary, it beats BPE, Unigram, and parity-aware BPE on fertility and MorphScore while keeping tokenization-cost disparity comparable. On four multilingual benchmarks, it posts aggregate gains of 1.48 to 2.00 points over those baselines. ArXiv · AI/CL/LG's note
The paper introduces a tokenizer that first finds reusable linguistic units, then builds the shared vocabulary. Tested across 104 languages with a 200K-token vocabulary, it beats BPE, Unigram, and parity-aware BPE on fertility and MorphScore while keeping tokenization-cost disparity comparable. On four multilingual benchmarks, it posts aggregate gains of 1.48 to 2.00 points over those baselines. ArXiv · AI/CL/LG's note
score 4