A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
A clean later hold-out still left pretrained forecasters advantaged where their training corpus looked familiar.
The paper tests 13 forecasting methods on post-release datasets meant to avoid direct contamination. Pretrained models win five of seven dataset groups, but stumble on one group against Theta and show no edge on daily exchange rates. The authors say seasonality and spectral entropy do not explain the pattern; corpus familiarity does. Their sharpest example is Wikipedia pageviews, where TimesFM’s reported pretraining mix closely matches the test domain and it posts the largest gain. ArXiv · AI/CL/LG's note
The paper tests 13 forecasting methods on post-release datasets meant to avoid direct contamination. Pretrained models win five of seven dataset groups, but stumble on one group against Theta and show no edge on daily exchange rates. The authors say seasonality and spectral entropy do not explain the pattern; corpus familiarity does. Their sharpest example is Wikipedia pageviews, where TimesFM’s reported pretraining mix closely matches the test domain and it posts the largest gain. ArXiv · AI/CL/LG's note
score 5