ML Newsnew | past | best | rss | submit about
Routing LLM queries using internal success predictions (70% cost reduction) (arxiv.org)
1 point by stansApprentice 234 days ago | hide | past | pdf | 3 comments on HN

In plain words: Before answering, a model's internal state hints whether it will succeed, and a simple readout of that signal beats question length or word patterns at predicting success. Routing questions to the right model beat the best single one while cutting cost up to 70%.

Abstract · LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations

Running LLMs with extended reasoning on every problem is expensive, but determining which inputs actually require additional compute remains challenging. We investigate whether their own likelihood of success is recoverable from their internal representations before generation, and if this signal can guide more efficient inference. We train linear probes on pre-generation activations to predict policy-specific success on math and coding tasks, substantially outperforming surface features such as question length and TF-IDF. Using E2H-AMC, which provides both human and model performance on identical problems, we show that models encode a model-specific notion of difficulty that is distinct from human difficulty, and that this distinction increases with extended reasoning. Leveraging these probes, we demonstrate that routing queries across a pool of models can exceed the best-performing model whilst reducing inference cost by up to 70\% on MATH, showing that internal representations enable practical efficiency gains even when they diverge from human intuitions about difficulty. Our code is available at: https://github.com/KabakaWilliam/llms_know_difficulty

William Lugoloobi, Thomas Foster, William Bankes, Chris Russell
arXiv:2602.09924 · cs.CL, cs.AI, cs.LG · submitted Feb 10, 2026 · updated Aug 11, 2026
abstract · pdf · html · Accepted at COLM 2026

add comment on HN

stansApprentice 234 days ago [–]
Author here. The core idea is pretty simple: train linear probes on the model's internal state before it generates anything to predict if it'll succeed. Then use those predictions to route queries: Send easy ones to cheap inference, hard ones to expensive reasoning.

Two findings that surprised us:

1. The same model has completely different internal representations of "difficulty" depending on decoding settings. What GPT-oss thinks is hard with greedy ≠ what it thinks is hard with sampling.

2. Model difficulty and human difficulty are orthogonal. The problems they struggle with aren't the ones we struggle with, and this gap increases with extended reasoning.

Code: https://github.com/KabakaWilliam/llms_know_difficulty Probes: https://huggingface.co/CoffeeGitta/pika-probes

Happy to answer questions.

reply
robinbanner 232 days ago [–]
This is a nice approach — using model internals to predict success before committing to full generation.

We took a different path at Komilion: classify the prompt upfront (regex fast path + lightweight LLM classifier) and route to the cheapest model that benchmarks well for that query type. Simpler, but works without model-specific training.

The 70% cost reduction figure matches what we see in practice. The insight that most queries are "easy" is the key — once you stop sending FAQ-level questions to frontier models, the savings are dramatic.

Curious if you have looked at combining both approaches — upfront classification for obvious cases, then internal state probes for the ambiguous ones in the middle.

reply
stansApprentice 229 days ago | parent [–]
Thank you!

I haven't used Komilion before, but it seems like we were thinking in the same direction. There's definitely room to add more to our suggested application of the probe. That's why we kept it simple so that people can add more business logic on top of how someone may want to route (so stuff like length, topic, etc)

reply
About | RSS | RSS (all) | HN arXiv