Not All Ranks Are Equal: Budget-Aware LoRA Merging Across Tasks
The paper says smarter rank allocation can narrow the gap between merged LoRAs and task-specific adapters.
The authors argue that current LoRA merging methods waste budget by giving every layer, and sometimes every task, the same rank. They introduce Net Utility, a data-free SVD-based scoring method that selects singular directions by estimated task value and cross-task interference. Applied across five merging methods and three merging spaces, it improves average performance by 2.1% on vision tasks and 2.2% on language tasks. HF Daily Papers' note
The authors argue that current LoRA merging methods waste budget by giving every layer, and sometimes every task, the same rank. They introduce Net Utility, a data-free SVD-based scoring method that selects singular directions by estimated task value and cross-task interference. Applied across five merging methods and three merging spaces, it improves average performance by 2.1% on vision tasks and 2.2% on language tasks. HF Daily Papers' note
score 4