The Structure of Quantization Damage in LLMs: Why the Next Bit Should Be Spent Globally
A small precision budget worked better when spread across quantization granularity than spent rescuing specific layers.
The paper tests nine open-weight models and finds that layer-level “damage” from post-training quantization is usually diffuse, not concentrated in obvious task circuits, compute-heavy areas, or weight statistics. In eight of nine models, recovering 75% of the lost accuracy required about half the layers. Across the group-128-compatible models, globally finer quantization beat local repair of the most recoverable layers by 21 to 52 points, even for the concentrated Qwen3-8B case. The authors argue that cheap correlates of damage are not enough; causal intervention is needed to see where restored precision actually helps. ArXiv · AI/CL/LG's note
The paper tests nine open-weight models and finds that layer-level “damage” from post-training quantization is usually diffuse, not concentrated in obvious task circuits, compute-heavy areas, or weight statistics. In eight of nine models, recovering 75% of the lost accuracy required about half the layers. Across the group-128-compatible models, globally finer quantization beat local repair of the most recoverable layers by 21 to 52 points, even for the concentrated Qwen3-8B case. The authors argue that cheap correlates of damage are not enough; causal intervention is needed to see where restored precision actually helps. ArXiv · AI/CL/LG's note
score 5