Enhancing Rubric-based RL via Self-Distillation
CriPO targets two rubric-RL failure modes: criteria never explored, and criteria learned but suppressed by aggregate rewards.
The paper says suppressed criteria appear in more than 57% of samples during training, averaging 1.8 per sample. CriPO uses on-policy self-distillation: one self-teacher injects missing criterion behavior, while another identifies useful criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages positive. In medicine and science benchmarks, the authors report stronger final performance than rubric-based RL with about half the optimization steps. HF Daily Papers' note
The paper says suppressed criteria appear in more than 57% of samples during training, averaging 1.8 per sample. CriPO uses on-policy self-distillation: one self-teacher injects missing criterion behavior, while another identifies useful criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages positive. In medicine and science benchmarks, the authors report stronger final performance than rubric-based RL with about half the optimization steps. HF Daily Papers' note
score 4