about
Do small language models know what they don't know? (arxiv.org)
3 points by Brajeshwar 11 days ago | hide | past | pdf | discuss on HN

In plain words: Small models' word-by-word confidence is useless: it stays near zero whether answers are right or wrong in 91% of cases. Sampling several answers and grouping by meaning finds uncertainty; sending them to a bigger model lifts accuracy by up to 50 percentage points.

Abstract

We explore whether entropy-based confidence signals can be leveraged to improve the accuracy of Small Language Models (SLMs) with fewer than 3 billion parameters, running entirely on consumer hardware. We evaluate seven distinct approaches, including token-level entropy early stopping, semantic entropy estimation, and uncertainty-aware routing to larger expert models, across 7 model pairs and 5 standard NLU benchmarks. Our key finding is that token-level entropy is effectively blind in SLMs: in 91% of dataset-model combinations, mean token entropy is near zero regardless of answer correctness, rendering token-based confidence signals unusable at this scale. We demonstrate that semantic entropy, computed by generating multiple samples, clustering answers by meaning, and measuring distributional uncertainty, recovers a viable confidence signal. Using semantic entropy to selectively route uncertain queries to a larger expert model yields accuracy improvements of up to +50 percentage points. Notably, cross-family routing (e.g., SmolLM 360M to Phi-3.5-mini) averages +22.0% improvement compared to +6.8% for same-family routing, revealing that expert model quality matters more than architectural compatibility. Our results suggest that the value proposition for entropy-based methods in SLMs is not computational savings but intelligent compute allocation: spending more tokens where they matter most.

Prashant Mudgal
arXiv:2609.20824 · cs.CL · submitted Jul 21, 2026
abstract · pdf · html · 9 pages, 8 figures

add comment on HN