Knowledge Distillation for Language Models

Yuqiao Wen, Freda Shi, Lili MouNAACL 2025

TL;DR

This paper proposes a tutorial on knowledge distillation (KD) for language models, aiming to make large, powerful models more accessible by transferring their knowledge to smaller, more efficient ones. It covers fundamental KD techniques, advanced methods like reinforcement learning-based KD and multi-teacher KD, and crucially, its applications to large language models (LLMs), including compression and self-distillation.

What Problem Does This Paper Address?

The explosive growth of large language models (LLMs) has brought unprecedented capabilities but also significant computational overhead and memory requirements. This makes them difficult to deploy and use for a wider audience. While techniques like pruning and quantization can reduce model size, knowledge distillation offers a complementary approach by transferring the "knowledge" from a large, capable "teacher" model to a smaller "student" model. This tutorial aims to provide a comprehensive understanding of KD techniques and their practical applications, particularly for LLMs, bridging the gap between foundational concepts and cutting-edge research.

Methods

The tutorial is structured into several parts:

  • Introduction and Overview: Sets the stage for KD and its importance.
  • KD Basics:
    • Intermediate-layer matching: Distilling knowledge from the internal representations (features and attention weights) of the teacher model. This includes discussing relational learning, which focuses on the relationships between features rather than their absolute values.
  • Prediction matching: Distilling the output predictions of the teacher model. This covers the classic cross-entropy approach and addresses the "multi-modality issue" where smaller students struggle to learn complex teacher distributions. Divergence-based methods are presented as solutions.
  • Advanced KD Techniques:
    • Reinforcement Learning (RL)-based KD: Leveraging RL to train student models, with a focus on deriving reward functions from the teacher model to align student behavior with human preferences and mitigate exposure bias.
  • Multi-teacher KD: Learning from multiple teachers, each with potentially different expertise. The "ensemble-then-distill" framework is highlighted as a way to consolidate knowledge before distillation.
  • KD Applications for LLMs:
    • LLM sequence compression: Reducing the length of prompts or generated sequences.
  • LLM self-distillation: Enabling LLMs to improve themselves by learning from their own outputs, leading to skills like instruction following and reasoning.
  • SOTA distilled systems: Examining practical examples of distilled LLMs like Alpaca, Vicuna, and DeepSeek.
  • Other applications: Including quantization, speculative decoding, and non-autoregressive translation.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Bridging the gap: KD is presented as a crucial technique for democratizing access to powerful LLMs by making them more efficient and deployable.
  • Beyond simple prediction matching: The tutorial emphasizes that effective KD involves more than just matching the final output; distilling intermediate representations and understanding relational structures are key.
  • Addressing LLM challenges: KD is not just for compression but also for imbuing smaller models with complex reasoning and instruction-following capabilities, and even enabling self-improvement in LLMs.
  • The multi-modality problem: A significant challenge in KD is when the teacher model's output distribution is too complex for a smaller student to learn, leading to issues like model collapse. The paper promises to discuss methods to mitigate this.
  • Practical relevance: The tutorial aims to connect theoretical KD concepts with real-world applications and state-of-the-art distilled LLM systems.

Key Results

Not discussed in the paper, as this is a tutorial proposal. The focus is on the content and structure of the tutorial rather than experimental results.

Limitations and Future Work

The paper does not explicitly state limitations of the proposed tutorial or future work within the tutorial's scope. However, the overarching goal of making LLMs more accessible implies that current limitations in LLM efficiency are the primary motivation. The tutorial's structure, moving from basics to advanced topics and applications, suggests a natural progression for future research in KD for LLMs.

Summary updated

BibTeX
@inproceedings{wen2025knowledge,
  author = {Yuqiao Wen and Freda Shi and Lili Mou},
  title = {{Knowledge Distillation for Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure