Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
The authors report up to 4.7 percentage points higher zero-shot accuracy with LEMS and a 19% average perplexity improvement with KFAC-SVD.
LEMS allocates compression ranks across layers using an error model that accounts for layer importance and error propagation. KFAC-SVD uses token-wise statistics to address rank collapse in earlier Fisher-based decomposition methods. Tests span Mistral, Qwen3 and Llama 3, with LEMS gains extending to models with 70 billion parameters.
HF Daily Papers' note
LEMS allocates compression ranks across layers using an error model that accounts for layer importance and error propagation. KFAC-SVD uses token-wise statistics to address rank collapse in earlier Fisher-based decomposition methods. Tests span Mistral, Qwen3 and Llama 3, with LEMS gains extending to models with 70 billion parameters.
HF Daily Papers' note
score 4