about
LLMs require curated context for reliable political fact-checking (arxiv.org)
3 points by teleforce 131 days ago | hide | past | pdf | discuss on HN

In plain words: Tested 15 chatbots on over 6,000 PolitiFact claims, comparing plain answers, extra reasoning, web search, and answers given hand-picked PolitiFact summaries. Reasoning barely helped and search only somewhat, but those curated summaries raised the average fact-checking score by 233%.

Abstract · Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Search

Large language models (LLMs) have raised hopes for automated end-to-end fact-checking, but prior studies report mixed results. As mainstream chatbots increasingly ship with reasoning capabilities and web search tools -- and millions of users already rely on them for verification -- rigorous evaluation is urgent. We evaluate 15 recent LLMs from OpenAI, Google, Meta, and DeepSeek on more than 6,000 claims fact-checked by PolitiFact, comparing standard models with reasoning- and web-search variants. Standard models perform poorly, reasoning offers minimal benefits, and web search provides only moderate gains, despite fact-checks being available on the web. In contrast, a curated RAG system using PolitiFact summaries improved macro F1 by 233% on average across model variants. These findings suggest that giving models access to curated high-quality context is a promising path for automated fact-checking.

Matthew R. DeVerna, Kai-Cheng Yang, Harry Yaojun Yan, Filippo Menczer
arXiv:2511.18749 · cs.CL, cs.CY, cs.IR · submitted Nov 24, 2025
abstract · pdf · html

add comment on HN