Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning
Agon trains two comparable models by making each beat the other after exposing its reasoning.
The paper argues that final-answer RL can reward longer traces without judging whether the reasoning improved. Agon alternates two models as drafter and reader-solver, rewarding each for out-solving a rival that has seen its work. At inference, the same pair runs as a two-stage cascade: one drafts, the other answers after reading. On DeepMath’s hard split with Qwen3, the author reports roughly double GRPO’s pass@1, with similar ordering on code tasks and other model families. HF Daily Papers' note
The paper argues that final-answer RL can reward longer traces without judging whether the reasoning improved. Agon alternates two models as drafter and reader-solver, rewarding each for out-solving a rival that has seen its work. At inference, the same pair runs as a two-stage cascade: one drafts, the other answers after reading. On DeepMath’s hard split with Qwen3, the author reports roughly double GRPO’s pass@1, with similar ordering on code tasks and other model families. HF Daily Papers' note
score 5