Statistical Inference for Rank Allocation in Low-Rank Adaptation
StatLoRA treats LoRA rank allocation as hypothesis testing, using p-values to decide which components survive a fixed rank budget.
The paper assigns each LoRA component a test statistic, then retains or prunes components under the prescribed budget. Its theory gives asymptotic normality for stochastic optimizer trajectories, including AdamW, and derives distributions for the component scores. Tests on DeBERTaV3-base, BART-Large, and Qwen2.5-7B show comparable or better results than vanilla LoRA, AdaLoRA, and IGU-LoRA at matched budgets. ArXiv · AI/CL/LG's note
The paper assigns each LoRA component a test statistic, then retains or prunes components under the prescribed budget. Its theory gives asymptotic normality for stochastic optimizer trajectories, including AdamW, and derives distributions for the component scores. Tests on DeBERTaV3-base, BART-Large, and Qwen2.5-7B show comparable or better results than vanilla LoRA, AdaLoRA, and IGU-LoRA at matched budgets. ArXiv · AI/CL/LG's note
score 4