Sky sphere representation in language models
Several ~100B open-source models appear to encode a decodable map of the night sky in their residual streams.
The authors test prompts asking what lies close to an object in the night sky. Most models studied show a sky-sphere representation, sometimes visible in top principal components. Leave-one-out tests find significant scores in all but one model, with reported median angular error as low as 12 to 21 degrees. The paper argues this is not just leakage from a correlated flat representation. ArXiv · AI/CL/LG's note
The authors test prompts asking what lies close to an object in the night sky. Most models studied show a sky-sphere representation, sometimes visible in top principal components. Leave-one-out tests find significant scores in all but one model, with reported median angular error as low as 12 to 21 degrees. The paper argues this is not just leakage from a correlated flat representation. ArXiv · AI/CL/LG's note
score 5