Beyond the Verdict: Agentic Code Judging with Verifiable Feedback
ARBITER is a 35B MoE verifier trained to find checkable counterexamples for bad competitive-programming submissions without spending official judging attempts.
The paper defines a judging task where the model sees only the problem statement and source code, then produces verdicts plus generated counterexamples as evidence. Its USACO-JUDGE benchmark uses 2026 USACO accepted and unaccepted programs from human experts and multiple LLMs. Reinforcement learning lifts ARBITER’s counterexample success on incorrect programs from 40.61% to 62.67%, and mean binary verdict accuracy from 71.41% to 87.41%. The authors also report gains in code selection and refinement, including an IOI 2026 case study where ARBITER-guided refinement pushes GPT-OSS-120B above the gold-medal threshold under the official submission limit. ArXiv · AI/CL/LG's note
The paper defines a judging task where the model sees only the problem statement and source code, then produces verdicts plus generated counterexamples as evidence. Its USACO-JUDGE benchmark uses 2026 USACO accepted and unaccepted programs from human experts and multiple LLMs. Reinforcement learning lifts ARBITER’s counterexample success on incorrect programs from 40.61% to 62.67%, and mean binary verdict accuracy from 71.41% to 87.41%. The authors also report gains in code selection and refinement, including an IOI 2026 case study where ARBITER-guided refinement pushes GPT-OSS-120B above the gold-medal threshold under the official submission limit. ArXiv · AI/CL/LG's note
score 5