Boosting LLM Exploration via Weak-Model Guidance in RLVR
Weak-model prefixes are used to keep RLVR-trained models from collapsing into narrower reasoning paths.
The paper says RLVR can improve reasoning while reducing policy entropy, hurting coverage when pass@$k$ is measured at larger k. Its method makes the target model continue from partial reasoning trajectories produced by a smaller, weaker model, using those unfamiliar starts to disrupt over-confident generation. Across mathematical benchmarks, the authors report consistent gains over vanilla RLVR, with larger improvements as k increases. They also say the approach avoids extra SFT, complex reward design, or elaborate prompting. ArXiv · AI/CL/LG's note
The paper says RLVR can improve reasoning while reducing policy entropy, hurting coverage when pass@$k$ is measured at larger k. Its method makes the target model continue from partial reasoning trajectories produced by a smaller, weaker model, using those unfamiliar starts to disrupt over-confident generation. Across mathematical benchmarks, the authors report consistent gains over vanilla RLVR, with larger improvements as k increases. They also say the approach avoids extra SFT, complex reward design, or elaborate prompting. ArXiv · AI/CL/LG's note
score 5