Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL
MRTPO reports faster convergence and better performance by directing each reward’s training signal to the tokens it concerns.
The method addresses a limitation of GDPO, which normalizes rewards separately but spreads each reward’s advantage evenly across the response. MRTPO replays the model’s response with and without information checked by each reward’s verifier, then uses differences in token probabilities to allocate credit. It requires no auxiliary models or token-level annotations. The authors report gains over the compared methods on text-to-SQL, multi-constraint instruction following and tool calling.
HF Daily Papers' note
The method addresses a limitation of GDPO, which normalizes rewards separately but spreads each reward’s advantage evenly across the response. MRTPO replays the model’s response with and without information checked by each reward’s verifier, then uses differences in token probabilities to allocate credit. It requires no auxiliary models or token-level annotations. The authors report gains over the compared methods on text-to-SQL, multi-constraint instruction following and tool calling.
HF Daily Papers' note
score 5