about
Towards Understanding Distilled Reasoning Models: A Representational Approach (arxiv.org)
3 points by bearseascape on Mar 6, 2025 | hide | past | pdf | discuss on HN

In plain words: They compared the inner workings of language models before and after training them on another model's worked-out reasoning, using a tool that finds shared and unique internal directions. Distilled models gained unique reasoning directions that can steer them toward overthinking or sharp, concise thinking.

Abstract

In this paper, we investigate how model distillation impacts the development of reasoning features in large language models (LLMs). To explore this, we train a crosscoder on Qwen-series models and their fine-tuned variants. Our results suggest that the crosscoder learns features corresponding to various types of reasoning, including self-reflection and computation verification. Moreover, we observe that distilled models contain unique reasoning feature directions, which could be used to steer the model into over-thinking or incisive-thinking mode. In particular, we perform analysis on four specific reasoning categories: (a) self-reflection, (b) deductive reasoning, (c) alternative reasoning, and (d) contrastive reasoning. Finally, we examine the changes in feature geometry resulting from the distillation process and find indications that larger distilled models may develop more structured representations, which correlate with enhanced distillation performance. By providing insights into how distillation modifies the model, our study contributes to enhancing the transparency and reliability of AI systems.

David D. Baek, Max Tegmark
arXiv:2503.03730 · cs.LG · submitted Mar 5, 2025 · updated Mar 25, 2025
abstract · pdf · html · 13 pages, 9 figures

add comment on HN