Understanding Reasoning from Pretraining to Post-Training
RL gains were predictable from pretraining loss in a controlled chess testbed.
The authors pretrained language models from 5M to 1B parameters on human chess games, then used supervised reasoning traces and RL on verifiable chess puzzles. They report that lower pretraining loss predicted stronger post-RL performance at a fixed RL compute budget, and that RL reward curves improved roughly linearly with more pretraining tokens. RL also behaved differently by puzzle difficulty: it boosted already-likely correct moves on easy puzzles, but surfaced barely present correct moves on hard ones. A smaller math-domain transfer test with a 1B model showed the same pattern: longer pretraining led to faster and higher RL gains. HF Daily Papers' note
The authors pretrained language models from 5M to 1B parameters on human chess games, then used supervised reasoning traces and RL on verifiable chess puzzles. They report that lower pretraining loss predicted stronger post-RL performance at a fixed RL compute budget, and that RL reward curves improved roughly linearly with more pretraining tokens. RL also behaved differently by puzzle difficulty: it boosted already-likely correct moves on easy puzzles, but surfaced barely present correct moves on hard ones. A smaller math-domain transfer test with a 1B model showed the same pattern: longer pretraining led to faster and higher RL gains. HF Daily Papers' note
score 5