A Margin Perspective on LoRA: Robustness to Catastrophic Forgetting and Adapter Merging (MaLoRA)
MaLoRA merges independently trained LoRA adapters without further training, using weights derived to balance task margins.
The paper links LoRA’s retention of prior learning and its ability to merge adapters through a margin-based analysis. Its theory covers multiclass linear classification with near-orthogonal tasks and ℓ₂ regularization. Under those assumptions, the authors show how regularization balances retention against adaptation and derive mixing weights that maximize the minimum margin across tasks. Experiments on vision and language classification tasks show MaLoRA matching or outperforming several adapter-merging baselines.
ArXiv · AI/CL/LG's note
The paper links LoRA’s retention of prior learning and its ability to merge adapters through a margin-based analysis. Its theory covers multiclass linear classification with near-orthogonal tasks and ℓ₂ regularization. Under those assumptions, the authors show how regularization balances retention against adaptation and derive mixing weights that maximize the minimum margin across tasks. Experiments on vision and language classification tasks show MaLoRA matching or outperforming several adapter-merging baselines.
ArXiv · AI/CL/LG's note
score 4