AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
AdviSD trains a small advisor to improve frozen frontier models by selectively learning from only the corrections most likely to matter.
The paper argues that learning from every plausible self-correction can hold an advisor back when some corrections do not meaningfully change the executor’s behavior. AdviSD combines outcome-based reinforcement learning with selective self-distillation, using the advisor’s own scoring of an executor response with and without its advice to choose supervision targets. In tests with Qwen3-8B advisors steering Gemini and Claude, it beats advisor-GRPO on BFCL-v3 and EnvScaler, and also outperforms random matched-count selection. The authors report that the trained advisors generalize beyond the training tasks and transfer across executor versions and model families. ArXiv · AI/CL/LG's note
The paper argues that learning from every plausible self-correction can hold an advisor back when some corrections do not meaningfully change the executor’s behavior. AdviSD combines outcome-based reinforcement learning with selective self-distillation, using the advisor’s own scoring of an executor response with and without its advice to choose supervision targets. In tests with Qwen3-8B advisors steering Gemini and Claude, it beats advisor-GRPO on BFCL-v3 and EnvScaler, and also outperforms random matched-count selection. The authors report that the trained advisors generalize beyond the training tasks and transfer across executor versions and model families. ArXiv · AI/CL/LG's note
score 4