Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following
Models can earn full instruction-following credit while providing little substantive answer content.
The authors find that training against the same verifiers can increase constraint satisfaction while reducing response quality in publicly released checkpoints. Their proposed fix, CALIF, adds a calibrated likelihood bonus using an existing frozen reference model, without an additional judge or learned reward model. Across models with 1.7 billion to 8 billion parameters, it improves IFBench constraint satisfaction while keeping response quality near or above pre-training-with-RL levels. It also beats the corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks.
HF Daily Papers' note
The authors find that training against the same verifiers can increase constraint satisfaction while reducing response quality in publicly released checkpoints. Their proposed fix, CALIF, adds a calibrated likelihood bonus using an existing frozen reference model, without an additional judge or learned reward model. Across models with 1.7 billion to 8 billion parameters, it improves IFBench constraint satisfaction while keeping response quality near or above pre-training-with-RL levels. It also beats the corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks.
HF Daily Papers' note
score 5