CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
CoRT reallocates GRPO’s single response-level advantage across tokens using rubric-dependent likelihood contrasts.
The paper targets a credit-assignment gap in rubric-based reinforcement learning, where structured judgments are usually collapsed into one reward and applied uniformly to every generated token. CoRT replays the same sampled response with and without the rubric-conditioned prompt, then uses tokenwise log-likelihood differences as weights for distributing the signed GRPO advantage. It does not add a learned token scorer or change the response-level reward. Across tested instruction-tuned models and reward granularities, the authors report an average 4.4-point gain over matched response-level GRPO. HF Daily Papers' note
The paper targets a credit-assignment gap in rubric-based reinforcement learning, where structured judgments are usually collapsed into one reward and applied uniformly to every generated token. CoRT replays the same sampled response with and without the rubric-conditioned prompt, then uses tokenwise log-likelihood differences as weights for distributing the signed GRPO advantage. It does not add a learned token scorer or change the response-level reward. Across tested instruction-tuned models and reward granularities, the authors report an average 4.4-point gain over matched response-level GRPO. HF Daily Papers' note
score 5