TTPO: Test-Time Policy Optimization
TTPO trains at test time without ground-truth labels by splitting rollouts into agreement and disagreement paths.
The paper argues that majority-vote pseudo-labels are risky because a wrong vote can corrupt the whole update. Its fix is asymmetric: distill rollouts that agree with the vote, and penalize disagreeing rollouts through grouped reinforcement learning. Token-level selection down-weights converged positions and targets confident errors. The authors report label-free results matching supervised OPSD on five competition-level benchmarks, including Qwen3-1.7B rising from 38.0% to 45.2% in test-time training. HF Daily Papers' note
The paper argues that majority-vote pseudo-labels are risky because a wrong vote can corrupt the whole update. Its fix is asymmetric: distill rollouts that agree with the vote, and penalize disagreeing rollouts through grouped reinforcement learning. Token-level selection down-weights converged positions and targets confident errors. The authors report label-free results matching supervised OPSD on five competition-level benchmarks, including Qwen3-1.7B rising from 38.0% to 45.2% in test-time training. HF Daily Papers' note
score 5