In-Place Tokenizer Expansion for Pre-trained LLMs
The paper claims an in-place tokenizer expansion can cut token counts sharply for added languages without losing the source checkpoint’s quality.
The method continues the existing BPE merges on multilingual data, preserving carried-over tokens and giving each new token an exact source-token decomposition. New embedding rows are initialized from the mean of their source sub-token embeddings, then adapted with embedding-only training followed by full-model continued pretraining. Applied to LFM2-8B-A1B, the expanded 128K tokenizer reduced Hindi and Vietnamese tokenization by about 2.4x and 2.6x, with Thai up to 4.0x. The authors estimate 2.2x to 3.7x per-character decode speedups for those languages on reference devices. ArXiv · AI/CL/LG's note
The method continues the existing BPE merges on multilingual data, preserving carried-over tokens and giving each new token an exact source-token decomposition. New embedding rows are initialized from the mean of their source sub-token embeddings, then adapted with embedding-only training followed by full-model continued pretraining. Applied to LFM2-8B-A1B, the expanded 128K tokenizer reduced Hindi and Vietnamese tokenization by about 2.4x and 2.6x, with Thai up to 4.0x. The authors estimate 2.2x to 3.7x per-character decode speedups for those languages on reference devices. ArXiv · AI/CL/LG's note
score 5