Normalized Low-Rank Adaptation
NoRA stabilizes LoRA by normalizing its down-projection matrices during training.
The paper argues that LoRA’s early training is dominated by the down-projection because the up-projection starts at zero. NoRA applies normalization there, and the authors say the same idea can also be used only at initialization to improve standard LoRA. Across pretraining, supervised finetuning, and reinforcement learning, they report faster convergence, better performance, improved stability, and less catastrophic forgetting. The method adds no trainable parameters and no inference-time compute. ArXiv · AI/CL/LG's note
The paper argues that LoRA’s early training is dominated by the down-projection because the up-projection starts at zero. NoRA applies normalization there, and the authors say the same idea can also be used only at initialization to improve standard LoRA. Across pretraining, supervised finetuning, and reinforcement learning, they report faster convergence, better performance, improved stability, and less catastrophic forgetting. The method adds no trainable parameters and no inference-time compute. ArXiv · AI/CL/LG's note
score 5