On Emergent Capabilities and Model Merging
Merged models keep shared hidden behavior, but do not reliably combine emergent capabilities.
The paper tests model merging on activation oracles and emergent-misaligned models across three model families. When both parent models carry broad misalignment, the merged model retains much of it across the mixing range. When only one parent has the emergent behavior, merging weakens it faster than the trained capability beside it. The authors also report that merging two single-task oracles does not reach the auditing ability of a jointly trained oracle. ArXiv · AI/CL/LG's note
The paper tests model merging on activation oracles and emergent-misaligned models across three model families. When both parent models carry broad misalignment, the merged model retains much of it across the mixing range. When only one parent has the emergent behavior, merging weakens it faster than the trained capability beside it. The authors also report that merging two single-task oracles does not reach the auditing ability of a jointly trained oracle. ArXiv · AI/CL/LG's note
score 5