When does text encoding matter for language modeling?
Compressive encodings significantly helped byte-level models for Chinese and Korean, but made little difference for Georgian and Tamil.
The study compared UTF-8 with other Unicode schemes, language-specific encodings and custom alternatives across four languages. The authors introduced Aether, which combines script-specific encodings with UTF-8 fallbacks to preserve full Unicode coverage. Benefits depended on how much information individual characters carry. The authors argue that encoding should be a language-specific design choice.
ArXiv · AI/CL/LG's note
The study compared UTF-8 with other Unicode schemes, language-specific encodings and custom alternatives across four languages. The authors introduced Aether, which combines script-specific encodings with UTF-8 fallbacks to preserve full Unicode coverage. Benefits depended on how much information individual characters carry. The authors argue that encoding should be a language-specific design choice.
ArXiv · AI/CL/LG's note
score 4