In plain words: While a chat session waits, the server runs the model ahead to the next choice point, so the next request starts warmed up. If the model is confident enough to answer outright, the first word lands in about 1 ms instead of 39 ms.
Abstract · Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path
A stateless inference server (vLLM, SGLang, TensorRT-LLM) idles between requests while the accelerator waits; a stateful session reclaims that idle time. Speculative pre-positioning decodes the session forward to its next decision point with the target model's own forward pass and no draft model, moving the cross-request prefill and entry-decode off the critical path: the next request resumes from a pre-paid entry on its delta, or, when a confidence gate fires, is answered from a cached distribution in one near-constant vocabulary scan with no decode, at a cost only of energy and a rare, bounded false accept. The payoff is conditional on capability: a capable model fires the gate at near-full coverage and about 87% precision (a smaller one never clears it), returning the first token in about 1.0 ms versus the 39 ms decode a prefix cache still pays.
Victor Norgren
arXiv:2606.29565 · cs.LG · submitted Jun 28, 2026
abstract · pdf · html