Cliff: Learning Process Rewards from the First Mistake
Cliff turns the first detected reasoning error into token-level reward shaping.
The paper argues that once a rollout goes wrong, later steps add little training signal because they depend on a bad prefix. Cliff uses an off-the-shelf LLM teacher to mark that first mistake, then rewards the correct prefix and penalizes the suffix. Across 12 scenarios, the authors report gains over on-policy distillation and standard GRPO, including with modest teachers. HF Daily Papers' note
The paper argues that once a rollout goes wrong, later steps add little training signal because they depend on a bad prefix. Cliff uses an off-the-shelf LLM teacher to mark that first mistake, then rewards the correct prefix and penalizes the suffix. Across 12 scenarios, the authors report gains over on-policy distillation and standard GRPO, including with modest teachers. HF Daily Papers' note
score 5