Normalized Low-Rank Adaptation
NoRA normalizes LoRA’s down-projection matrices to stabilize and speed adaptation.
The paper argues LoRA’s early training is dominated by the down-projection because the up-projection starts at zero. NoRA applies normalization during training, and the authors say a one-time initialization-only version can also improve standard LoRA. They report faster convergence, better performance, greater stability, and less catastrophic forgetting across pretraining, supervised finetuning, and reinforcement learning. The method adds no trainable parameters and no inference-time cost. HF Daily Papers' note
The paper argues LoRA’s early training is dominated by the down-projection because the up-projection starts at zero. NoRA applies normalization during training, and the authors say a one-time initialization-only version can also improve standard LoRA. They report faster convergence, better performance, greater stability, and less catastrophic forgetting across pretraining, supervised finetuning, and reinforcement learning. The method adds no trainable parameters and no inference-time cost. HF Daily Papers' note
score 5