Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters
Hindcast tries to test LLM forecasting without letting models peek at the future.
The paper says standard backtests can leak answers through post-event retrieval or newer training data. Hindcast instead replays resolved Polymarket questions from a chosen past cutoff, limiting retrieval to Reddit posts written before that date. It scores the model against both the real outcome and the market price at the cutoff. Retrieval still helps most models when Reddit had relevant prior discussion, but hurts when the archive was mostly speculation. ArXiv · AI/CL/LG's note
The paper says standard backtests can leak answers through post-event retrieval or newer training data. Hindcast instead replays resolved Polymarket questions from a chosen past cutoff, limiting retrieval to Reddit posts written before that date. It scores the model against both the real outcome and the market price at the cutoff. Retrieval still helps most models when Reddit had relevant prior discussion, but hurts when the archive was mostly speculation. ArXiv · AI/CL/LG's note
score 5