Disentangling Representation Evolution in Transformers through Directional Decomposition

The paper argues that transformer updates can be split into direction-preserving and direction-changing parts, and that split exposes where interventions hold up.

The authors decompose learned updates into parallel and perpendicular components across pretrained models. They find meaningful parallel components beyond the residual identity path. In their experiments, manipulating the exclude-self value-space parallel component is more robust than comparable residual-space or perpendicular edits. The same lens separates compression errors and improves from-scratch pretraining when full-aggregate parallel suppression is used, especially in value space. HF Daily Papers' note