Megadose The AI curve, compressed.

Reading or Remembering? Measuring Temporal Contamination in Language Models

· ArXiv · AI/CL/LG ·
Models recalled past economic outcomes, but the study found no forecasting advantage that survived correction for multiple testing.

The authors compared modern open models with models trained only on text predating a fixed cutoff, using FOMC statements and minutes from 2000–2026. Even models unable to know the outcomes scored up to 0.64 AUC from dates alone, making chance an unsuitable baseline for measuring memory. Membership-inference scores tracked document era and type rather than actual inclusion in training data. The authors recommend auditing memory directly against models without access to future information on the same questions.

ArXiv · AI/CL/LG's note

score 5

Categories: Research