How Proper Scoring Rules Shape LLM Forecasting
The paper finds that “proper” scoring rules can train LLM forecasters into measurably different error patterns.
The authors compare five proper scoring rules as reward functions for binary forecasts on resolved real-world events. Models trained under different rules vary in calibration, probability use, and estimated bias, information, and noise, even when aggregate accuracy is close. Brier training gives the best observed Brier score and AUC-ROC; log training gives the best observed log score and lowest calibration error. The paper cautions that each condition used one seed, so some differences may be stochastic. ArXiv · AI/CL/LG's note
The authors compare five proper scoring rules as reward functions for binary forecasts on resolved real-world events. Models trained under different rules vary in calibration, probability use, and estimated bias, information, and noise, even when aggregate accuracy is close. Brier training gives the best observed Brier score and AUC-ROC; log training gives the best observed log score and lowest calibration error. The paper cautions that each condition used one seed, so some differences may be stochastic. ArXiv · AI/CL/LG's note
score 4