Task Context Shapes Component Roles in Language Models
The paper argues that the same attention heads can play different roles depending on task context.
The authors test fixed model components on identical graph facts across five open-weight models. They measure how interventions change attention heads, later activations, and final answers. In Gemma-2-9B and Qwen3-14B, an activation edit shifts which fact the model reads even when the written instruction is unchanged. Their response maps suggest task-specific context better predicts later head behavior than an averaged account.
Source: ArXiv · AI/CL/LG's note
The authors test fixed model components on identical graph facts across five open-weight models. They measure how interventions change attention heads, later activations, and final answers. In Gemma-2-9B and Qwen3-14B, an activation edit shifts which fact the model reads even when the written instruction is unchanged. Their response maps suggest task-specific context better predicts later head behavior than an averaged account.
Source: ArXiv · AI/CL/LG's note
score 4