about
Leaderboard Hacking: Preference-Based Model Evals Vulnerable to Manipulation (arxiv.org)
1 point by Betelbuddy 3 days ago | hide | past | pdf | discuss on HN

In plain words: Arena leaderboards rank models by pooling anonymous head-to-head votes on user prompts. A dishonest provider can game that tally with strategic votes, favorable prompts, or inserted dummy models—each lifts a model 15 to 20 places, and all three up to 45.

Abstract · Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation

Arena-style rankings of language models are a widely used evaluation framework in online leaderboards, research papers, and model evaluation in production. These rankings aim to realistically assess model performance using pairwise comparisons of anonymous models on user prompts, aggregating votes via the Bradley-Terry model to produce the final ranking. However, we find that these leaderboards are highly vulnerable to several manipulation strategies that unscrupulous providers could use to boost a model's rank: strategic voting (individual votes that boost a model's rank, including votes on irrelevant models), strategic prompting (choosing prompts that favor a model), and strategic nomination (boosting the rank of a model by inserting irrelevant models). We demonstrate these effects on public leaderboard data, analyze the circumstances that cause these vulnerabilities, and describe approaches to safeguard against each attack. Notably, these issues hold for any preference-based evaluation using the Bradley-Terry model that allows open prompting, voting, or model submission. These manipulations boost the rank of nearly every tested model; each attack can boost a model by up to 15-20 places in a ranking and, combined, they can boost a model by up to 45 places.

Eve Fleisig, Joachim Baumann, Dylan Hadfield-Menell, Dirk Hovy
arXiv:2611.14769 · cs.AI · submitted Nov 10, 2026
abstract · pdf · html

add comment on HN