about
LLMs Outperform Experts on Challenging Biology Benchmarks (arxiv.org)
1 point by belter on May 12, 2025 | hide | past | pdf | discuss on HN

In plain words: Twenty-seven AI models were tested ten times each on eight biology exams covering genetics, cloning, and viruses. On the hardest virology test, the newest model now answers twice as well as expert virologists, far better than earlier versions.

Abstract

This study systematically evaluates 27 frontier Large Language Models on eight biology benchmarks spanning molecular biology, genetics, cloning, virology, and biosecurity. Models from major AI developers released between November 2022 and April 2025 were assessed through ten independent runs per benchmark. The findings reveal dramatic improvements in biological capabilities. Top model performance increased more than 4-fold on the challenging text-only subset of the Virology Capabilities Test over the study period, with OpenAI's o3 now performing twice as well as expert virologists. Several models now match or exceed expert-level performance on other challenging benchmarks, including the biology subsets of GPQA and WMDP and LAB-Bench CloningScenarios. Contrary to expectations, chain-of-thought did not substantially improve performance over zero-shot evaluation, while extended reasoning features in o3-mini and Claude 3.7 Sonnet typically improved performance as predicted by inference scaling. Benchmarks such as PubMedQA and the MMLU and WMDP biology subsets exhibited performance plateaus well below 100%, suggesting benchmark saturation and errors in the underlying benchmark data. The analysis highlights the need for more sophisticated evaluation methodologies as AI systems continue to advance.

Lennart Justen
arXiv:2505.06108 · cs.LG, cs.AI, q-bio.QM · submitted May 9, 2025 · updated May 21, 2025
abstract · pdf

add comment on HN