Not All Thinking is Created Equal: Latent Reasoning Discovers a Recurrent Search Algorithm for Depth Generalization
Latent-reasoning models generalized to longer-hop problems by learning a recurrent forward-search circuit.
The paper compares five GPTNeoX variants trained from scratch on ProsQA-Ext. Vanilla, chain-of-thought, and pause-token models did well in distribution but leaned on local graph features and failed more often at greater depths. The latent variants showed internal dynamics consistent with reachability propagation, and the bottleneck version exposed a sparse circuit for relation retrieval, state updates, and candidate matching. ArXiv · AI/CL/LG's note
The paper compares five GPTNeoX variants trained from scratch on ProsQA-Ext. Vanilla, chain-of-thought, and pause-token models did well in distribution but leaned on local graph features and failed more often at greater depths. The latent variants showed internal dynamics consistent with reachability propagation, and the bottleneck version exposed a sparse circuit for relation retrieval, state updates, and candidate matching. ArXiv · AI/CL/LG's note
score 6