Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off
A penalty encouraging LoRA adapters to use more of their available capacity lowered mean Wiki perplexity at every tested rank of 16 or higher across three models.
The authors find that standard LoRA updates often have an effective rank far below their configured rank, especially for high-rank adapters. Their method adds a Rényi-2 effective-rank regularizer to the language-modeling loss without requiring singular value decomposition. In mixed-domain tests with multiple random seeds, a penalty weight of 0.01 improved mean training perplexity or increased it by at most 0.02 in all but one model-rank comparison. Separate single-task and scheduling experiments showed that the balance between learning and retention still depends on the configuration.
ArXiv · AI/CL/LG's note
The authors find that standard LoRA updates often have an effective rank far below their configured rank, especially for high-rank adapters. Their method adds a Rényi-2 effective-rank regularizer to the language-modeling loss without requiring singular value decomposition. In mixed-domain tests with multiple random seeds, a penalty weight of 0.01 improved mean training perplexity or increased it by at most 0.02 in all but one model-rank comparison. Separate single-task and scheduling experiments showed that the balance between learning and retention still depends on the configuration.
ArXiv · AI/CL/LG's note
score 5