Inducing Process Supervision from Outcome-Only Reinforcement Learning
TIPS trains step-level reward behavior using only final-answer labels.
The paper introduces Thinking-Induced Process Supervision, where a model generates reasoning, step labels, and an outcome label, but reinforcement learning rewards only the correctness of the final outcome. The authors argue that useful intermediate checking can emerge because better step verification helps the model judge the outcome correctly. They test the method on math and agent benchmarks across four backbone families. One reported model, TIPS-Qwen3-4B-Thinking-2507, reaches 85.2 F1 on ProcessBench using 3.2K outcome-labeled trajectories, beating the evaluated trained PRMs and prompt-only judges listed in the abstract while still trailing o1-mini. HF Daily Papers' note
The paper introduces Thinking-Induced Process Supervision, where a model generates reasoning, step labels, and an outcome label, but reinforcement learning rewards only the correctness of the final outcome. The authors argue that useful intermediate checking can emerge because better step verification helps the model judge the outcome correctly. They test the method on math and agent benchmarks across four backbone families. One reported model, TIPS-Qwen3-4B-Thinking-2507, reaches 85.2 F1 on ProcessBench using 3.2K outcome-labeled trajectories, beating the evaluated trained PRMs and prompt-only judges listed in the abstract while still trailing o1-mini. HF Daily Papers' note
score 5