What Breaks Under Pruning in Smart Homes, and When? Evaluating LLM Degradation Across Architectures and Task Complexity
Pruned dense LLMs lost reliable smart-home tool calling much sooner than MoE models.
The paper tests four model architectures and several pruning methods on more than 19,500 smart-home tool-calling cases after supervised fine-tuning. It finds dense models had only narrow safe pruning ranges before performance dropped sharply, while mixture-of-experts models tolerated more pruning. The first failures were in grounded details like device, argument, and value, before the models lost the broader action intent. In aggressive dense pruning, the authors also saw systematic over-refusal. ArXiv · AI/CL/LG's note
The paper tests four model architectures and several pruning methods on more than 19,500 smart-home tool-calling cases after supervised fine-tuning. It finds dense models had only narrow safe pruning ranges before performance dropped sharply, while mixture-of-experts models tolerated more pruning. The first failures were in grounded details like device, argument, and value, before the models lost the broader action intent. In aggressive dense pruning, the authors also saw systematic over-refusal. ArXiv · AI/CL/LG's note
score 4