about
SimpleQA Verified: Reliable Factuality Benchmark to Measure Parametric Knowledge (arxiv.org)
3 points by handfuloflight 290 days ago | hide | past | pdf | discuss on HN

In plain words: A cleaned-up set of 1,000 short-answer questions tests how well AI models recall facts from memory, fixing wrong labels, repeated questions, and skewed topics in the original. Gemini 2.5 Pro scored 55.6, beating GPT-5 and other top models.

Abstract · SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

We introduce SimpleQA Verified, a 1,000-prompt benchmark for evaluating Large Language Model (LLM) short-form factuality based on OpenAI's SimpleQA. It addresses critical limitations in OpenAI's benchmark, including noisy and incorrect labels, topical biases, and question redundancy. SimpleQA Verified was created through a rigorous multi-stage filtering process involving de-duplication, topic balancing, and source reconciliation to produce a more reliable and challenging evaluation set, alongside improvements in the autorater prompt. On this new benchmark, Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6, outperforming other frontier models, including GPT-5. This work provides the research community with a higher-fidelity tool to track genuine progress in parametric model factuality and to mitigate hallucinations. The benchmark dataset, evaluation code, and leaderboard are available at: https://www.kaggle.com/benchmarks/deepmind/simpleqa-verified.

Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein, Dipanjan Das
arXiv:2509.07968 · cs.CL · submitted Sep 9, 2025 · updated Mar 10, 2026
abstract · pdf · html

add comment on HN
Also discussed: Sep 2025 (3 points, 0 comments)