MixKD: Towards Efficient Distillation of Large-scale Language Models

Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, Lawrence Carin

TL;DR

MixKD is a new framework that supercharges knowledge distillation for large language models by using "mixup" data augmentation. By training a smaller student model to mimic a large teacher's behavior on interpolated data points, MixKD achieves superior generalization and performance, particularly when task-specific training data is scarce.

What Problem Does This Paper Address?

Large-scale language models like BERT have revolutionized NLP, but their massive size makes them slow and expensive to deploy on edge devices or in low-resource environments. Knowledge Distillation (KD) is the standard solution: a large "teacher" model trains a smaller "student" model to mimic its outputs.

However, standard KD has a major bottleneck: the student only learns from the teacher on the exact samples present in the training set. If the task-specific dataset is small, the student has limited "questions" to ask the teacher, leading to overfitting and poor generalization. The authors identify a need for a way to probe the teacher's knowledge in the "empty spaces" between real data points to help the student learn a more robust representation.

Methods

The authors introduce MixKD, which integrates the mixup augmentation strategy into the distillation process.

  1. Mixup for NLP: Unlike images, text is discrete. MixKD performs mixup by linearly interpolating the word embeddings of two different sentences at each time step. To handle varying sentence lengths, they use simple zero-padding.
  2. The Distillation Objective: The student doesn't just learn from the teacher's predictions on real data. It is also encouraged to match the teacher's output for these "virtual" interpolated samples.
  3. Hybrid Loss: The final training objective is a weighted sum of three parts:
  4. Standard cross-entropy on original data.
  5. Cross-entropy for the student on mixup samples.
  6. Distillation loss (mimicking the teacher) on mixup samples.

The paper also provides a theoretical proof showing that this approach reduces the gap between generalization error and empirical error, essentially guaranteeing a more robust model under certain conditions.

Datasets

The researchers evaluated MixKD on the GLUE (General Language Understanding Evaluation) benchmark, a standard suite of diverse NLP tasks including:

  • Sentiment Analysis: SST-2
  • Paraphrase Detection: MRPC, QQP
  • Natural Language Inference: MNLI, QNLI, RTE

To test the method's robustness, they also created limited-data versions of these sets by downsampling them to just 10% and 1% of their original size.

Key Insights and Ideas

  • Probing the Teacher: The authors liken MixKD to a student who learns more by asking the teacher "what if?" questions about hypothetical scenarios (interpolated data) rather than just memorizing textbook examples.
  • Manifold Alignment: Visualization of the latent space (Figure 2) shows that MixKD forces interpolated samples to align with the natural data manifold. This helps the student explore meaningful regions of the feature space that standard fine-tuning ignores.
Latent space visualization: (a) standard fine-tuning vs (b) MixKD. Triangles represent interpolated 'mixup' neighbors.
  • Efficiency over Complexity: Compared to other augmentation methods like TinyBERT (which uses a complex conditional BERT generator), MixKD uses a simple linear interpolation that is computationally much cheaper while yielding better results.
  • Additivity: MixKD is complementary to other techniques. The authors found that combining MixKD with Back-Translation (BT)—translating text to another language and back—yielded the best overall performance.

Key Results

MixKD consistently outperformed standard KD and other competitive compression baselines like DistilBERT and Patient Knowledge Distillation (PKD).

  • GLUE Performance: A 6-layer MixKD student achieved 92.09% accuracy on SST-2, capturing 99.88% of the performance of the 12-layer BERT teacher while being significantly faster.
  • Data Scarcity: The gains were most dramatic in limited-data settings. With only 1% of the training data, MixKD improved accuracy by up to 4.1% over standard fine-tuning on tasks like QNLI.
Figure 1: Results of limited data case, where both the teacher and student models are learned with only 10% (left) or 1% of the training data (right).
  • Efficiency: As shown in Table 1, the MixKD variants (especially those combined with Back-Translation) set new state-of-the-art marks for 3-layer and 6-layer BERT models across almost all GLUE tasks.
Table 1: GLUE dev set results comparing various student model configurations.

Limitations and Future Work

While highly effective, the current study focuses on task-specific distillation (fine-tuning for a specific goal). The authors note that MixKD could potentially be extended to the pre-training stage of language models, which could lead to even more powerful general-purpose compressed models. They also suggest that incorporating more advanced, recent mixup variants could further narrow the gap between student and teacher performance.

Summary updated

Abstract

Large-scale language models have recently demonstrated impressive empirical performance. Nevertheless, the improved results are attained at the price of bigger models, more power consumption, and slower inference, which hinder their applicability to low-resource (both memory and computation) platforms. Knowledge distillation (KD) has been demonstrated as an effective framework for compressing such big models. However, large-scale neural network systems are prone to memorize training instances, and thus tend to make inconsistent predictions when the data distribution is altered slightly. Moreover, the student model has few opportunities to request useful information from the teacher model when there is limited task-specific data available. To address these issues, we propose MixKD, a data-agnostic distillation framework that leverages mixup, a simple yet efficient data augmentation approach, to endow the resulting model with stronger generalization ability. Concretely, in addition to the original training examples, the student model is encouraged to mimic the teacher's behavior on the linear interpolation of example pairs as well. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the generalization error and the empirical error. To verify its effectiveness, we conduct experiments on the GLUE benchmark, where MixKD consistently leads to significant gains over the standard KD training, and outperforms several competitive baselines. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
BibTeX
@inproceedings{liang2021mixkd,
  author = {Kevin J Liang and Weituo Hao and Dinghan Shen and Yufan Zhou and Weizhu Chen and Changyou Chen and Lawrence Carin},
  title = {{MixKD: Towards Efficient Distillation of Large-scale Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=UFGEelJkLu5},
}
Figure