Not All Reasoning Generalizes: A Case Study on Learning Generalizable Reasoning Strategies in RLVR
The model learned step-by-step derivation, then abandoned it for brute force when the tasks changed.
In a controlled string-equation benchmark, reinforcement learning with verifiable rewards produced high accuracy on tasks within the training distribution. When evaluation required more solutions or longer outputs, the model switched to ineffective enumeration and accuracy fell sharply. Co-training on bijective inversion tasks substantially restored derivation and accuracy, even though those tasks were not consistently harder or closer to the evaluation setting. The authors argue that outcome-only rewards can influence which reasoning strategies a model uses without explicit supervision of its intermediate steps.
ArXiv · AI/CL/LG's note
In a controlled string-equation benchmark, reinforcement learning with verifiable rewards produced high accuracy on tasks within the training distribution. When evaluation required more solutions or longer outputs, the model switched to ineffective enumeration and accuracy fell sharply. Co-training on bijective inversion tasks substantially restored derivation and accuracy, even though those tasks were not consistently harder or closer to the evaluation setting. The authors argue that outcome-only rewards can influence which reasoning strategies a model uses without explicit supervision of its intermediate steps.
ArXiv · AI/CL/LG's note
score 5