The Geometry of Inference in Transformer Residual Streams
The paper argues that transformer residual states become geometrically specific to their eventual outputs early, even when simple distance barely moves.
Across six pretrained language models, the authors compare intermediate residual states with their own final states and with final states from other contexts. The “own” endpoint becomes favored over the average alternative early, while some individual competing endpoints can stay closer until later layers. They separate norm, directional alignment, and endpoint geometry, showing why competitor counts can drop sharply without a matching Euclidean-distance change. Entries of new competitors rule out a simple straight-line path to the final state. HF Daily Papers' note
Across six pretrained language models, the authors compare intermediate residual states with their own final states and with final states from other contexts. The “own” endpoint becomes favored over the average alternative early, while some individual competing endpoints can stay closer until later layers. They separate norm, directional alignment, and endpoint geometry, showing why competitor counts can drop sharply without a matching Euclidean-distance change. Entries of new competitors rule out a simple straight-line path to the final state. HF Daily Papers' note
score 4