Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning
Residual routing helps most when paired with LoRA, not as a clean replacement for it.
The paper tests manifold-constrained hyper-connections as a PEFT method by wrapping frozen OLMo-2 Transformers with learned residual-routing modules. It finds mHC can finetune frozen models, but fixing the residual mixing matrix to identity often works better in this setting. Alone, mHC does not consistently beat LoRA. At matched trainable-parameter budgets, mHC plus LoRA improves language-modeling loss and produces task-dependent benchmark gains at 1B and 7B scale. ArXiv · AI/CL/LG's note
The paper tests manifold-constrained hyper-connections as a PEFT method by wrapping frozen OLMo-2 Transformers with learned residual-routing modules. It finds mHC can finetune frozen models, but fixing the residual mixing matrix to identity often works better in this setting. Alone, mHC does not consistently beat LoRA. At matched trainable-parameter budgets, mHC plus LoRA improves language-modeling loss and produces task-dependent benchmark gains at 1B and 7B scale. ArXiv · AI/CL/LG's note
score 4