TL;DR
Large language models (LLMs) are powerful but computationally expensive. This paper systematically compares and analyzes several "task-agnostic" knowledge distillation methods for compressing these models. The authors find that Multi-Head Attention (MHA) transfer, particularly using the MiniLMv2 approach, generally yields the best performance, offering a compelling alternative to Hidden State (HS) transfer, while Output Distribution (OD) transfer lags behind.
What Problem Does This Paper Address?
Modern NLP relies heavily on large Transformer-based language models that achieve state-of-the-art results across numerous tasks. However, their sheer size and computational demands make them impractical for real-world deployment, especially in latency-sensitive applications. Knowledge distillation offers a solution by transferring the "knowledge" from a large teacher model to a smaller, more efficient student model. This paper focuses on task-agnostic distillation, where a single student model is trained to be effective across various downstream tasks without task-specific fine-tuning. The authors aim to provide a comprehensive comparison of leading task-agnostic distillation methods to guide practitioners and researchers in selecting the most effective approach.
Methods
The paper investigates three primary categories of task-agnostic distillation:
- Output Distribution (OD) Transfer: The student model tries to mimic the probability distribution of the teacher model's final output. This is a classic distillation technique.
- Hidden State (HS) Transfer: The student model is trained to replicate the intermediate hidden states of the teacher model. A key challenge here is determining how to map student layers to teacher layers, and the authors explore various strategies (single mapping, 1-to-1, and 1-to-N mappings).
- Multi-Head Attention (MHA) Transfer: This method focuses on distilling the attention mechanisms within Transformer layers. The authors compare two approaches:
- MiniLMv2: Distills attention relation matrices (e.g., Q-Q, K-K, V-V) derived from concatenated query, key, and value projections.
- DirectMiniLM: A novel variant proposed by the authors, which directly distills the query, key, and value mappings using linear transformations. This method is designed to be more directly comparable to HS transfer and to shed light on MiniLMv2's success.
All methods were reproduced on a unified codebase for fair comparison.
Datasets
- Training:
- Monolingual: English Wikipedia and BookCorpus (for distilling BERT).
- Multilingual: A subset of the CC100 dataset (for distilling XLM-ROBERTa).
- Evaluation:
- Monolingual: English GLUE benchmark (excluding CoLA).
- Multilingual: XNLI dataset (across 15 languages).
Key Insights and Ideas
- MHA Transfer Reigns Supreme: Multi-Head Attention transfer methods, particularly MiniLMv2, consistently outperform other approaches across various student architectures and language settings. This suggests that distilling the core attention mechanism is more effective for general-purpose compression than just mimicking outputs or hidden states.
- HS Transfer is a Solid Baseline: Hidden State transfer remains a competitive option, especially when employing sophisticated layer mapping strategies (like 1-to-N). It offers a good balance of performance and implementation simplicity.
- OD Transfer Struggles: Classical Output Distribution transfer, even when initialized from a pre-distilled checkpoint, generally lags behind HS and MHA transfer. This highlights the difficulty of learning nuanced representations from output probabilities alone, especially for complex models and general-purpose objectives.
- Layer Mapping Matters for HS: The effectiveness of HS transfer is highly dependent on how student layers are mapped to teacher layers. More complex mappings (1-to-N) tend to yield better results, particularly in multilingual settings.
- DirectMiniLM Illuminates MiniLMv2: The proposed DirectMiniLM method provides a clearer understanding of why MiniLMv2 works well by offering a more direct comparison to HS transfer. The authors find that MiniLMv2 and DirectMiniLM perform comparably, with MiniLMv2 showing a slight edge on XNLI, potentially due to its ability to capture relational representations.
Key Results
- MHA Transfer Dominance: MHA transfer methods (MiniLMv2 and DirectMiniLM) achieved the best average GLUE scores (e.g., 84.4% for 6L-DistilBERT) and XNLI scores (e.g., 69.1% for 6L-DistilBERT) among the distilled models.
- HS Transfer Competitiveness: HS transfer with the "Uniform+Last" strategy achieved strong results, comparable to MHA transfer in some monolingual settings (e.g., 84.1% for 6L-DistilBERT on GLUE).
- OD Transfer Underperformance: OD transfer consistently showed lower performance across all student sizes and settings compared to HS and MHA transfer.
- Student Model Performance: The paper evaluated distillation on student models ranging from 3 to 6 layers, demonstrating significant compression with minimal performance loss (e.g., a 4L student achieving 93% of the teacher's performance).
- Multilingual Gains: Sophisticated layer mapping strategies for HS transfer proved particularly beneficial in the multilingual setting.
Limitations and Future Work
- Architecture Constraints: The study focuses on architecture-agnostic distillation, meaning the student and teacher can have different architectural parameters. However, the authors acknowledge that they primarily used encoder-only models and did not explore distillation for decoder-only or encoder-decoder architectures.
- Hyperparameter Tuning: For efficiency, extensive grid searches for fine-tuning hyperparameters were not conducted. The authors used a common set of parameters after some manual tuning.
- Future Work: The authors suggest exploring more advanced distillation techniques, such as scheduled annealing, multi-stage distillation with teacher assistants, and meta-learning for optimizing distillation. They also recommend further investigation into the differences between MiniLMv2 and DirectMiniLM, particularly regarding the transfer of relational versus absolute position-wise representations.
Summary updated