LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
A 5B-parameter model was trained only on an 88B-token elementary-school curriculum to make its knowledge limits observable.
The authors introduce LITTLECURRICULUM, a pretraining corpus built around U.S. Grade 5-and-under material while excluding higher-grade concepts, facts, and vocabulary. Training from scratch on it produces LITTLELEARNER, meant as a sandbox for studying what a model can acquire under a defined exposure regime. In early experiments, post-training and in-context learning helped the model use knowledge it already had, but did not extend it into out-of-scope capabilities. ArXiv · AI/CL/LG's note
The authors introduce LITTLECURRICULUM, a pretraining corpus built around U.S. Grade 5-and-under material while excluding higher-grade concepts, facts, and vocabulary. Training from scratch on it produces LITTLELEARNER, meant as a sandbox for studying what a model can acquire under a defined exposure regime. In early experiments, post-training and in-context learning helped the model use knowledge it already had, but did not extend it into out-of-scope capabilities. ArXiv · AI/CL/LG's note
score 5