about
Breaking the Tokenizer Barrier: On-Policy Distillation Across Model Families (arxiv.org)
2 points by Jimmc414 97 days ago | hide | past | pdf | discuss on HN

In plain words: A token-mapping step lets a teacher and student that split text into different word pieces share full training signals, instead of the usual practice of just copying the teacher's finished answers. This trained students more efficiently than that copying approach across many tests.

Abstract · Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models. However, existing OPD distillation methods require teacher and student models to share the same tokenizer, restricting the applicability of OPD within the model series. Current mainstream practice typically employs Supervised Fine-Tuning (SFT) on teacher-generated responses for cross-tokenizer distillation, which fails to capture the rich knowledge embedded in the teacher's probability distribution. In this work, we enable the standard on-policy distillation method to operate across model families, ensuring that high-fidelity token-level signals can propagate across different tokenizers with a precise token-mapping algorithm. Extensive experiments show that cross-tokenizer OPD is significantly more compute-efficient than baselines on various benchmarks. Our results unlock a broader range of teacher-student pairs for OPD, opening up new avenues for adapting and enhancing interactions between LLMs.

Yifan Niu, Han Xiao, Dongyi Liu, Zelong Wang, Dihong Gong, Yasheng Wang, Jia Li
arXiv:2606.09456 · cs.LG · submitted Jun 8, 2026
abstract · pdf · html

add comment on HN