Megadose The AI curve, compressed.

Structure--Residual Inductive Biases for Language Model Pretraining

· ArXiv · AI/CL/LG ·
Combining two representation biases reduced GPT-2 Medium’s negative log-likelihood by 1.44% after 300 million training tokens.

Boundary Bottleneck Concept routes information from token segments through boundary states, while Continuous Concept separates low-frequency structure from high-frequency residual through representation regularization. BBC worked best in early layers; CC was most effective in later layers. The authors report that gains persisted in training runs up to 20 billion tokens. Diagnostics showed greater predictive importance for boundary states with BBC and lower spectral residual energy with CC.

ArXiv · AI/CL/LG's note

score 4

Categories: Research