Goal Encoding in LLMs
Transplanting an internal sub-goal encoding made a language model perform the same operation in another programming language, using different tokens.
In a coding setting with multiple sub-tasks, researchers located the next sub-goal’s encoding in a small set of attention heads. They could decode upcoming actions from those heads in natural language, and interventions showed the encoding causally influenced the next action. The encoding persisted throughout execution. The authors propose that the model repeatedly rereads the task specification and checks its progress against the growing context, an account they test with targeted causal interventions.
ArXiv · AI/CL/LG's note
In a coding setting with multiple sub-tasks, researchers located the next sub-goal’s encoding in a small set of attention heads. They could decode upcoming actions from those heads in natural language, and interventions showed the encoding causally influenced the next action. The encoding persisted throughout execution. The authors propose that the model repeatedly rereads the task specification and checks its progress against the growing context, an account they test with targeted causal interventions.
ArXiv · AI/CL/LG's note
score 5