about
LLMs tested on historical prediction markets, NONE outperformed the market (arxiv.org)
4 points by dataMike 13 days ago | hide | past | pdf | 2 comments on HN

In plain words: A practice arena rebuilds 1,568 resolved prediction-market events as day-by-day news and question steps, so AI forecasters can be tested and trained the same way repeatedly. Training a smaller model on a bigger one's traces lifted its accuracy from 34.8% to 42.8%.

Abstract · Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Forecasting requires large language model (LLM) agents to gather evolving evidence, revise their beliefs, and make calibrated predictions over time. Yet evaluating and improving these capabilities is challenging because real-world events unfold continuously and outcomes may take months to resolve. We introduce Forecast-Dojo, a replayable environment that reconstructs resolved prediction-market events as sequences of dated forecasting steps, enabling reproducible evaluation, longitudinal analysis, and training from agent interactions. It comprises 1,568 Polymarket events split temporally into training and evaluation sets, alongside 18.8 million dated news articles. Across 12 models, research tools consistently improve forecasting performance, and greater capture of new evidence is associated with larger improvements over time. Nevertheless, all models trail historical market forecasts, while carrying a belief notebook across dates reduces research costs without consistently improving forecast quality. We supervise fine-tuned Qwen3-30B-A3B-Thinking-2507 on collected trajectories from Qwen3-235B-A22B-Thinking-2507 to demonstrate the environment's training utility. The student's accuracy is improved from 34.8% to 42.8% on the evaluation set.

Liqin Ye, Haorui Wang, Fardin Ahmed, Rongzhi Zhang, Yuan He, Ziyuan Lin, Yanbin Yin, Jing Peng, Michael Galarnyk, Sudheer Chava, Chao Zhang
arXiv:2609.28876 · cs.AI, cs.LG · submitted Sep 24, 2026 · updated Oct 26, 2026
abstract · pdf · html

add comment on HN

Is an LLM even the right tool for the job here?

Prediction market historic data will mostly be tabular. Each outcome will have a number of discreet features associated with it, natural or engineered.

Wouldn't a gradient boosted tree approach work better in this domain? xgboost, lightgbm etc.

Or, if a neural network must be used, then one specifically tuned for tabular data work, like TabPFN, might be a better choice.

Using an LLM here just feels like the wrong tool choice.

[1] https://github.com/PriorLabs/tabpfn

That reminds me of a story--probably an HN submission--where someone tuned a model on historical data which got insanely close to predicting exactly what would happen every day. The "real" vs "predicted" graphs were almost indistinguishable.

Unfortunately, the reason they were so similar was that it decided each day's stock prices would be the same as the end of yesterday, which was on some measures very close yet also useless for prophetic profits.