A Broader Look at Model Merging: Rethinking Implicit Regularization Induced by Task Arithmetic
The paper argues that the usual coefficient search in model merging may be holding models back.
The authors say task-arithmetic merging confines candidates to the subspace spanned by task-specific weight updates. In their experiments, optimizing merged weights outside that constraint improves common merging methods across architectures, domains, and a one-example-per-class setting. They also report that directly optimizing pretrained weights can beat some existing merging methods. The takeaway is a narrower one: better multi-task weights may sit outside the standard task-arithmetic search space. ArXiv · AI/CL/LG's note
The authors say task-arithmetic merging confines candidates to the subspace spanned by task-specific weight updates. In their experiments, optimizing merged weights outside that constraint improves common merging methods across architectures, domains, and a one-example-per-class setting. They also report that directly optimizing pretrained weights can beat some existing merging methods. The takeaway is a narrower one: better multi-task weights may sit outside the standard task-arithmetic search space. ArXiv · AI/CL/LG's note
score 5