Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR
In a small Transformer experiment, removing a shared parameter direction prevented collapse; restoring it brought collapse back.
The paper studies label-free reinforcement learning that rewards a model for matching its own majority answer. Reasoning initially improves, but self-assessed reward can approach one while accuracy falls, sometimes leaving the same answer for every problem. Its mathematical model attributes collapse to shared perturbations that grow faster than individual problems settle on answers as the training set expands. Simulations support that mechanism and predict no collapse for sufficiently confident initial models.
ArXiv · AI/CL/LG's note
The paper studies label-free reinforcement learning that rewards a model for matching its own majority answer. Reasoning initially improves, but self-assessed reward can approach one while accuracy falls, sometimes leaving the same answer for every problem. Its mathematical model attributes collapse to shared perturbations that grow faster than individual problems settle on answers as the training set expands. Simulations support that mechanism and predict no collapse for sufficiently confident initial models.
ArXiv · AI/CL/LG's note
score 5