WMIP: A Working-Memory-Inspired Programmable Probe of Cumulative State Tracking in Large Language Models
Across 18 LLMs, accuracy fell from 84.9% on single-step arithmetic to 31.7% when tracking three to seven updates.
WMIP tests cumulative state tracking within a single prompt, using program-computed answers and sequences of 3 to 100 updates. At 50 updates and beyond, no model exceeded 10% accuracy. Core accuracy reached 88.0% with a larger budget and instructions to write out steps and put the answer last; the gains came mainly from the instruction changes. The results show that output instructions substantially affect measured accuracy.
HF Daily Papers' note
WMIP tests cumulative state tracking within a single prompt, using program-computed answers and sequences of 3 to 100 updates. At 50 updates and beyond, no model exceeded 10% accuracy. Core accuracy reached 88.0% with a larger budget and instructions to write out steps and put the answer last; the gains came mainly from the instruction changes. The results show that output instructions substantially affect measured accuracy.
HF Daily Papers' note
score 5