In plain words: Instead of asking ChatGPT to predict 2022 events directly, they had it write fictional future stories where characters recount what happened after its training data ended. Story prompts guessed more accurately than direct questions, and accuracy soared when the answers were in its training data.
Abstract · Can Base ChatGPT be Used for Forecasting without Additional Optimization?
This study investigates whether OpenAI's ChatGPT-3.5 and ChatGPT-4 can forecast future events. To evaluate the accuracy of the predictions, we take advantage of the fact that the training data at the time of our experiments (mid 2023) stopped at September 2021, and ask about events that happened in 2022. We employed two prompting strategies: direct prediction and what we call future narratives which ask ChatGPT to tell fictional stories set in the future with characters retelling events that happened in the past, but after ChatGPT's training data had been collected. We prompted ChatGPT to engage in storytelling, particularly within economic contexts. After analyzing 100 trials, we find that future narrative prompts significantly enhanced ChatGPT-4's forecasting accuracy. This was especially evident in its predictions of major Academy Award winners as well as economic trends, the latter inferred from scenarios where the model impersonated public figures like the Federal Reserve Chair, Jerome Powell. As a falsification exercise, we repeated our experiments in May 2024 at which time the models included more recent training data. ChatGPT-4's accuracy significantly improved when the training window included the events being prompted for, achieving 100% accuracy in many instances. The poorer accuracy for events outside of the training window suggests that in the 2023 prediction experiments, ChatGPT-4 was forming predictions based solely on its training data. Narrative prompting also consistently outperformed direct prompting. These findings indicate that narrative prompts leverage the models' capacity for hallucinatory narrative construction, facilitating more effective data synthesis and extrapolation than straightforward predictions. Our research reveals new aspects of LLMs' predictive capabilities and suggests potential future applications in analytical contexts.
Van Pham, Scott Cunningham
arXiv:2404.07396 · econ.GN, cs.AI · submitted Apr 11, 2024 · updated Jul 4, 2024
abstract · pdf · html · 77 pages, added falsification exercises in section `Post Scriptum:...' with new figures; new title 61 pages, 26 figures; corrected typos
The possibility I'd like to see eliminated before paying much attention to this is simply that they've given it additional training data newer than 2021 without making that public.
> The backbone of the current forefront of LLMs is an architecture called pre-generative transformers, or GPT.
Hmm!