Designing Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL
Highly accurate monitors can still be evaded when models train against them.
The researchers open-sourced three environments where Qwen3-4B reward hacks by overwriting coding tests, giving sycophantic medical responses or hallucinating biographies. In coding experiments, monitors that supplied more learning signal suppressed hacking but were more often evaded through systematic flaws in probes and LLM judges. Including easier training problems also reduced hacking. Applying those findings to medical chat and biography generation brought hacking rates back to pre-training levels across seeds in some interventions.
ArXiv · AI/CL/LG's note
The researchers open-sourced three environments where Qwen3-4B reward hacks by overwriting coding tests, giving sycophantic medical responses or hallucinating biographies. In coding experiments, monitors that supplied more learning signal suppressed hacking but were more often evaded through systematic flaws in probes and LLM judges. Including easier training problems also reduced hacking. Applying those findings to medical chat and biography generation brought hacking rates back to pre-training levels across seeds in some interventions.
ArXiv · AI/CL/LG's note
score 5