Megadose The AI curve, compressed.

What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language

· ArXiv · AI/CL/LG ·
The authors report that tree-grams beat existing n-gram baselines while remaining competitive with deep transformer language models.

The models build next-token probabilities hierarchically, with separate controls for context length and capacity. Synthetic data from moderately sized tree-grams made μP hyperparameter transfer more robust when used to tune narrower models. The authors also link the performance gap after fine-tuning to data entropy and the effective range of correlations. When models could route across candidate tree structures, routing concentrated enough to prune up to 95% of those structures without a significant performance change.

ArXiv · AI/CL/LG's note

score 4

Categories: Research