In plain words: A test of 300-plus prompts where the ideal reply is short, measuring how many extra characters a chatbot adds beyond a minimal baseline answer. Across 76 chatbots, median extra length varied tenfold, with models padding ambiguous questions and adding explanations to one-line coding tasks.
Abstract · Do Chatbot LLMs Talk Too Much? The YapBench Benchmark
Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini increasingly act as general-purpose copilots, yet they often respond with unnecessary length on simple requests, adding redundant explanations, hedging, or boilerplate that increases cognitive load and inflates token-based inference cost. Prior work suggests that preference-based post-training and LLM-judged evaluations can induce systematic length bias, where longer answers are rewarded even at comparable quality. We introduce YapBench, a lightweight benchmark for quantifying user-visible over-generation on brevity-ideal prompts. Each item consists of a single-turn prompt, a curated minimal-sufficient baseline answer, and a category label. Our primary metric, YapScore, measures excess response length beyond the baseline in characters, enabling comparisons across models without relying on any specific tokenizer. We summarize model performance via the YapIndex, a uniformly weighted average of category-level median YapScores. YapBench contains over three hundred English prompts spanning three common brevity-ideal settings: (A) minimal or ambiguous inputs where the ideal behavior is a short clarification, (B) closed-form factual questions with short stable answers, and (C) one-line coding tasks where a single command or snippet suffices. Evaluating 76 assistant LLMs, we observe an order-of-magnitude spread in median excess length and distinct category-specific failure modes, including vacuum-filling on ambiguous inputs and explanation or formatting overhead on one-line technical requests. We release the benchmark and maintain a live leaderboard for tracking verbosity behavior over time.
Vadim Borisov, Michael Gröger, Mina Mikhael, Richard H. Schreiber
arXiv:2601.00624 · cs.LG · submitted Jan 2, 2026
abstract · pdf · html
Here is the top 10 right now:
I also checked how the Claude models did specifically: No other Anthropic model seems to be there for now. I would have been very curious to see the more recent ones.