Disentangling Representation Evolution in Transformers through Directional Decomposition
The paper splits transformer updates into direction-preserving and direction-changing components, then shows that split matters for editing, compression, and training.
He, Zhang, and Yan decompose learned updates into parallel and perpendicular parts across pretrained transformer models. They report sizable parallel components beyond the residual identity path, and find that manipulating exclude-self value-space parallel components is more robust than comparable residual-space or perpendicular edits. The same lens separates compression errors, with perpendicular error distinguishing methods more clearly than parallel error. In from-scratch pretraining, suppressing full-aggregate parallel components lowers validation-loss trajectories and improves downstream averages, strongest in value space. ArXiv · AI/CL/LG's note
He, Zhang, and Yan decompose learned updates into parallel and perpendicular parts across pretrained transformer models. They report sizable parallel components beyond the residual identity path, and find that manipulating exclude-self value-space parallel components is more robust than comparable residual-space or perpendicular edits. The same lens separates compression errors, with perpendicular error distinguishing methods more clearly than parallel error. In from-scratch pretraining, suppressing full-aggregate parallel components lowers validation-loss trajectories and improves downstream averages, strongest in value space. ArXiv · AI/CL/LG's note
score 4