Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
RSTG tries to recover gradients in RLVR cases where GRPO gets no useful learning signal.
The paper says GRPO can lose gradients when every response in a group gets the same reward, and naive on-policy distillation can make performance worse. RSTG limits teacher guidance to failed zero-variance prompts, weights samples by teacher confidence, and targets tokens where the student is uncertain or far from the teacher. It also adds supervised training on correct teacher-generated trajectories to supply positive signals. The authors report gains over naive GRPO+OPD of 4.02% on math and 3.05% on code. HF Daily Papers' note
The paper says GRPO can lose gradients when every response in a group gets the same reward, and naive on-policy distillation can make performance worse. RSTG limits teacher guidance to failed zero-variance prompts, weights samples by teacher confidence, and targets tokens where the student is uncertain or far from the teacher. It also adds supervised training on correct teacher-generated trajectories to supply positive signals. The authors report gains over naive GRPO+OPD of 4.02% on math and 3.05% on code. HF Daily Papers' note
score 5