MInTRL: Off-policy Intervention can boost On-policy RL
MInTRL keeps RL mostly on-policy while using brief judge corrections to widen exploration.
The method lets a judge-intervention policy inspect a rollout, replace wrong suffixes with short fixes, then hand generation back to the current policy. Training uses a sequence-level advantage-regression objective, avoiding importance sampling. The authors report gains over standard on-policy and off-policy baselines on math and code benchmarks. Ablations say the approach still works with self-intervention and different judges, with best results at moderate intervention levels. HF Daily Papers' note
The method lets a judge-intervention policy inspect a rollout, replace wrong suffixes with short fixes, then hand generation back to the current policy. Training uses a sequence-level advantage-regression objective, avoiding importance sampling. The authors report gains over standard on-policy and off-policy baselines on math and code benchmarks. Ablations say the approach still works with self-intervention and different judges, with best results at moderate intervention levels. HF Daily Papers' note
score 4