Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMs
Few-shot examples made models use internal “world” representations more directly for prediction.
The paper tests graph-tracking setups across six models from four families and finds prediction improves when demonstrations come from different generated worlds. Linear probes show graph information encoded in hidden states, with few-shot demonstrations shifting those representations into new, nearly orthogonal directions. Interventions on those shifted representations hurt performance more than interventions on other subspaces. The authors report similar gains on ARC-AGI-1&2, web-agent tasks, and Othello. ArXiv · AI/CL/LG's note
The paper tests graph-tracking setups across six models from four families and finds prediction improves when demonstrations come from different generated worlds. Linear probes show graph information encoded in hidden states, with few-shot demonstrations shifting those representations into new, nearly orthogonal directions. Interventions on those shifted representations hurt performance more than interventions on other subspaces. The authors report similar gains on ARC-AGI-1&2, web-agent tasks, and Othello. ArXiv · AI/CL/LG's note
score 5