DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging
DARTS targets hidden-state drift that appears when fine-tuned decoder LLMs are merged.
The paper says decoder models need different correction than encoder models because causal masking lets representation bias build up across token positions. Its method weights correction more heavily at high-entropy positions, where generation decisions are more fragile. It also adds a per-position bias term while keeping the parameter cost near 0.1% of the model. The authors report gains over standard surgery on HumanEval, GSM8K, and AlpacaEval using Llama-2-7B models. ArXiv · AI/CL/LG's note
The paper says decoder models need different correction than encoder models because causal masking lets representation bias build up across token positions. Its method weights correction more heavily at high-entropy positions, where generation decisions are more fragile. It also adds a per-position bias term while keeping the parameter cost near 0.1% of the model. The authors report gains over standard surgery on HumanEval, GSM8K, and AlpacaEval using Llama-2-7B models. ArXiv · AI/CL/LG's note
score 5