Abstract
Knowledge distillation (KD) is widely used for compressing a teacher model to reduce its inference cost and memory footprint, by training a smaller student model. However, current KD methods for auto-regressive sequence models suffer from distribution mismatch between output sequences seen during training and those generated by the student during inference. To address this issue, we introduce Generalized Knowledge Distillation (GKD). Instead of solely relying on a fixed set of output sequences, GKD trains the student on its self-generated output sequences by leveraging feedback from the teacher on such sequences. Unlike supervised KD approaches, GKD also offers the flexibility to employ alternative loss functions between the student and teacher, which can be useful when the student lacks the expressivity to mimic the teacher's distribution. Furthermore, GKD facilitates the seamless integration of distillation with RL fine-tuning (RLHF). We demonstrate the efficacy of GKD for distilling auto-regressive language models on summarization, translation, and arithmetic reasoning tasks, and task-agnostic distillation for instruction-tuning.
Introduction
In the race to make Large Language Models (LLMs) faster and smaller, Knowledge Distillation (KD) has become a primary strategy. The goal is to train a compact "student" model to mimic the behavior of a massive "teacher" model. However, auto-regressive models face a fundamental challenge: distribution mismatch, also known as exposure bias.
Standard distillation typically uses "Teacher Forcing," where the model is trained only on ground-truth prefixes. At inference time, however, the student must generate text based on its own previous predictions. If the student makes a single error, it enters a state it never encountered during training, leading to cascading failures. GKD addresses this by allowing students to learn from their own self-generated mistakes under the guidance of the teacher.
Methodology: Generalized Knowledge Distillation
GKD shifts the training paradigm from "off-policy" (using fixed datasets) to "on-policy" (using student-generated data). The framework introduces three key innovations:
- On-Policy Sampling: The student generates its own output sequences during training. This ensures the training distribution matches the actual distribution the model will encounter during inference.
- Teacher Feedback: For every token in the student's generated sequence, the teacher model provides the target probability distribution, offering corrective feedback on the student's specific output.
- Divergence Flexibility: GKD allows for alternative loss functions like Reverse KL or Generalized Jensen-Shannon Divergence (JSD). While standard Forward KL is "mean-seeking," Reverse KL is "mode-seeking," which is often more effective when a small student lacks the capacity to capture the full complexity of the teacher's distribution.
Experimental Results
The researchers evaluated GKD across a diverse range of tasks using T5 and Flan-T5 models (ranging from 77M to 3B parameters), including summarization (XSum), translation (WMT14), and arithmetic reasoning (GSM8K).
As shown in the figure above, GKD consistently outperformed prevalent KD baselines. Compared to standard Supervised KD, GKD achieved relative performance gains of 2.1x on summarization, 1.7x on translation, and 1.9x on arithmetic reasoning.
Data Efficiency
GKD demonstrates remarkable data efficiency. In the XSum summarization task, the on-policy feedback loop allowed the model to learn much faster than traditional methods.
Notably, GKD trained on just 5% of the data outperformed standard Supervised KD trained on 100% of the data. This suggests that high-quality feedback on the student's own generated distribution is significantly more valuable than a larger volume of static, off-policy data.
Key Insights
- Addressing Exposure Bias: By training on-policy, GKD directly mitigates the train-inference gap that plagues auto-regressive models.
- Instruction Tuning: On general benchmarks like MMLU and BBH, GKD provided a 1-2% absolute accuracy boost over supervised distillation, proving its utility for task-agnostic instruction following.
- RL Integration: GKD facilitates the integration of distillation with Reinforcement Learning from AI Feedback (RLAIF). Using the distillation loss as a regularizer helps maintain model quality and reduce hallucinations while optimizing for specific rewards.
- Self-Distillation: Even when the student and teacher are the same size, GKD improves performance, suggesting the on-policy feedback loop helps models refine their own reasoning processes.
Conclusion
Generalized Knowledge Distillation (GKD) provides a robust framework for compressing auto-regressive models by solving the distribution mismatch problem. By leveraging on-policy sampling and flexible divergence measures, GKD enables the creation of efficient students that retain the high performance of their larger teachers. Future work may extend these principles to other modalities such as audio, video, and image generation.
Summary updated