Which Skill to Distill? SGUID: Selecting a Compact Skill Bank for Model-Skill Co-Evolution
SGUID finds that a small filtered skill set can beat or match distilling the whole bank.
The paper says fewer than 25% of retrieved skills gave useful distillation signals in on-policy training. SGUID keeps only skills that consistently produce effective learning signals, then distills those into the model. Across Olmo and Qwen models, 6 selected skills matched or exceeded full-bank distillation on three of four models, despite full banks being up to 11x larger. A second round selected 3 new skills and improved Qwen3-8B from 64.3% to 66.3%, while unfiltered updates hurt Qwen3-4B on HMMT25. Source: ArXiv · AI/CL/LG's note.
The paper says fewer than 25% of retrieved skills gave useful distillation signals in on-policy training. SGUID keeps only skills that consistently produce effective learning signals, then distills those into the model. Across Olmo and Qwen models, 6 selected skills matched or exceeded full-bank distillation on three of four models, despite full banks being up to 11x larger. A second round selected 3 new skills and improved Qwen3-8B from 64.3% to 66.3%, while unfiltered updates hurt Qwen3-4B on HMMT25. Source: ArXiv · AI/CL/LG's note.
score 4