In plain words: They trained AI forecasters with five reward formulas that all pay off for honest probability guesses on real-world events. The models had similar accuracy but different kinds of errors, and each did best on the exact score it was trained to optimize.
Abstract · How Proper Scoring Rules Shape LLM Forecasting
This paper evaluates how reward function choice shapes the performance and behavior of LLM forecasters. We compare five proper scoring rules as training objectives for binary forecasts of resolved real-world events. Although the rules share the same theoretical incentive for truthful probability reporting, the resulting models differ in calibration, probability use, and estimated profiles of bias, information, and noise, with smaller differences in aggregate accuracy and discrimination. The Brier-trained model has the lowest observed Brier score and highest AUC-ROC, while the log-trained model has the highest observed log score and lowest calibration error. Models with similar aggregate performance also reach that performance through different combinations of bias, information, and noise. Proper scoring rules therefore need not behave interchangeably as training objectives. Reward choice may shape not only how well an LLM forecasts, but how its forecasting errors are structured.
Benjamin Turtel, Paul Wilczewski, Kris Skotheim, Ville A. Satopää, Philip E. Tetlock
arXiv:2608.28482 · cs.LG, cs.AI · submitted Aug 28, 2026 · updated Sep 9, 2026
abstract · pdf · html · Added a three-seed replication of the Log and Brier reward conditions, including forecasting metrics and BIN analysis