What's in a Tree ? Hierarchical $n$-gram models as controlled approximators of language
The authors report that tree-grams beat existing n-gram baselines while remaining competitive with deep transformer language models.
The models build next-token probabilities hierarchically, with separate controls for context length and capacity. Synthetic data from moderately sized tree-grams made μP hyperparameter transfer more robust when used to tune narrower models. The authors also link the performance gap after fine-tuning to data entropy and the effective range of correlations. When models could route across candidate tree structures, routing concentrated enough to prune up to 95% of those structures without a significant performance change.
ArXiv · AI/CL/LG's note
The models build next-token probabilities hierarchically, with separate controls for context length and capacity. Synthetic data from moderately sized tree-grams made μP hyperparameter transfer more robust when used to tune narrower models. The authors also link the performance gap after fine-tuning to data entropy and the effective range of correlations. When models could route across candidate tree structures, routing concentrated enough to prune up to 95% of those structures without a significant performance change.
ArXiv · AI/CL/LG's note
score 4