A Riemannian Geometry for Low-rank Adaptation
The paper treats LoRA’s redundant parameterization as geometry, then uses that geometry to precondition training steps.
The authors define a Riemannian metric for the quotient manifold induced by equivalent LoRA factors. Their preconditioner is designed so LoRA’s weight update stays closest to the full fine-tuning gradient within LoRA’s allowed first-order update space. They also claim each updated weight matrix is closer in Frobenius norm to full fine-tuning than conventional or unpreconditioned LoRA. Experiments are reported on language and vision fine-tuning tasks. ArXiv · AI/CL/LG's note
The authors define a Riemannian metric for the quotient manifold induced by equivalent LoRA factors. Their preconditioner is designed so LoRA’s weight update stays closest to the full fine-tuning gradient within LoRA’s allowed first-order update space. They also claim each updated weight matrix is closer in Frobenius norm to full fine-tuning than conventional or unpreconditioned LoRA. Experiments are reported on language and vision fine-tuning tasks. ArXiv · AI/CL/LG's note
score 5