In plain words: Small language models trained to mimic a bigger model's answers were compared with plain-trained and commercial ones on skill and computing cost. The distilled model matched models much bigger while using over 2,000 times less computing to build than plain training.
Abstract · Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings
Knowledge distillation offers a transformative pathway to developing powerful, yet efficient, small language models (SLMs) suitable for resource-constrained environments. In this paper, we benchmark the performance and computational cost of distilled models against their vanilla and proprietary counterparts, providing a quantitative analysis of their efficiency. Our results demonstrate that distillation creates a superior performance-tocompute curve. We find that creating a distilled 8B model is over 2,000 times more compute-efficient than training its vanilla counterpart, while achieving reasoning capabilities on par with, or even exceeding, standard models ten times its size. These findings validate distillation not just as a compression technique, but as a primary strategy for building state-of-the-art, accessible AI
Sachin Gopal Wani, Eric Page, Ajay Dholakia, David Ellison
arXiv:2602.20164 · cs.CL, cs.LG · submitted Jan 28, 2026
abstract · pdf · 16 pages, 5 figures, accepted at the the 2025 TPCTC Conference