ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
ARCHead targets the LM-head left uncompressed by common weight-only quantization stacks, cutting its persistent storage by 3.7-3.9x without keeping a dense BF16 copy.
The paper combines a quantized low-rank core, group-wise INT4 residuals, and an activation-metric low-rank correction for the output projection. On Qwen3-8B-Base, it reports 25.6% of BF16 head storage with 1.007 relative perplexity, while storage-matched naive INT4 lands at 1.14-1.16. Used after AWQ or bitsandbytes, replacing the remaining BF16 head adds 0.006-0.007 cross-entropy with under 2% throughput change in the authors’ measurements. HF Daily Papers' note
The paper combines a quantized low-rank core, group-wise INT4 residuals, and an activation-metric low-rank correction for the output projection. On Qwen3-8B-Base, it reports 25.6% of BF16 head storage with 1.007 relative perplexity, while storage-matched naive INT4 lands at 1.14-1.16. Used after AWQ or bitsandbytes, replacing the remaining BF16 head adds 0.006-0.007 cross-entropy with under 2% throughput change in the authors’ measurements. HF Daily Papers' note
score 4