In plain words: A small model learns from a powerful paid AI whose inner workings are hidden, using a middle model to help pass along the knowledge. This beat both the usual hidden-teacher training and the standard approach that peeks inside the teacher's brain.
Abstract · Knowledge Distillation of Black-Box Large Language Models
Given the exceptional performance of proprietary large language models (LLMs) like GPT-4, recent research has increasingly focused on boosting the capabilities of smaller models through knowledge distillation (KD) from these powerful yet black-box teachers. While leveraging the high-quality outputs of these teachers is advantageous, the inaccessibility of their internal states often limits effective knowledge transfer. To overcome this limitation, we introduce Proxy-KD, a novel method that uses a proxy model to facilitate the efficient transfer of knowledge from black-box LLMs to smaller models. Our experiments show that Proxy-KD not only enhances the performance of KD from black-box teacher models but also surpasses traditional white-box KD techniques.~This approach presents a compelling new avenue for distilling knowledge from advanced LLMs.
Hongzhan Chen, Ruijun Chen, Yuqi Yi, Xiaojun Quan, Chenliang Li, Ming Yan, Ji Zhang
arXiv:2401.07013 · cs.CL · submitted Jan 13, 2024 · updated Nov 9, 2024
abstract · pdf · html
Related paper that's a good read: https://arxiv.org/abs/1908.08962