Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation
Hints rescued code-generation failures, but most of those fixes were already reachable without the hint.
The paper tests Qwen2.5-3B-Instruct and Phi-3.5-mini on HumanEval+ and MBPP+ with executable evaluation. Relevant hints helped, but unrelated hints and extra unhinted sampling also recovered many of the same cases. Mechanistic tests found no clear net accuracy gain from adding the shared hint-related activation direction. Full textual specifications worked far better than the tested virtual-KV prefixes on context-defined problems. ArXiv · AI/CL/LG's note
The paper tests Qwen2.5-3B-Instruct and Phi-3.5-mini on HumanEval+ and MBPP+ with executable evaluation. Relevant hints helped, but unrelated hints and extra unhinted sampling also recovered many of the same cases. Mechanistic tests found no clear net accuracy gain from adding the shared hint-related activation direction. Full textual specifications worked far better than the tested virtual-KV prefixes on context-defined problems. ArXiv · AI/CL/LG's note
score 4