Patient Knowledge Distillation for BERT Model Compression

Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu

Abstract

Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: (i) PKD-Last: learning from the last k layers; and (ii) PKD-Skip: learning from every k layers. These two patient distillation schemes enable the exploitation of rich information in the teacher's hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multi-layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training efficiency, without sacrificing model accuracy.

Introduction

Large-scale pre-trained models like BERT have revolutionized NLP, but their massive size—ranging from 110 million parameters in BERT-Base to 340 million in BERT-Large—presents significant challenges for deployment on edge devices and real-time production environments. While Knowledge Distillation (KD) is a standard technique for model compression, traditional "vanilla" KD often focuses solely on the teacher's final output (logits). This approach limits the student's ability to capture the complex internal reasoning of the teacher, often leading to suboptimal generalization.

To address this, we introduce Patient Knowledge Distillation (PKD). By requiring the student model to "patiently" mimic the teacher's internal representations across multiple intermediate layers, we enable the student to capture the foundational logic of the larger model, resulting in a more robust and efficient lightweight architecture.

Methodology

The core of PKD is the Patient Teacher (PT) Loss, a Mean Square Error (MSE) loss that aligns the student's hidden states with those of the teacher. To maintain efficiency, we specifically match the representations of the [CLS] token, which serves as BERT's summary of sentence-level semantics.

We propose two primary distillation strategies to map teacher layers to student layers:

  1. PKD-Last: The student learns from the final k layers of the teacher.
  2. PKD-Skip: The student learns from every k-th layer (e.g., layers 2, 4, 6, 8, and 10 of a 12-layer teacher).
Architectural overview of Patient Knowledge Distillation (PKD) strategies.

The total training objective combines this PT loss with standard cross-entropy loss (from gold labels) and standard distillation loss (from the teacher's soft probability outputs).

Experimental Setup

We evaluated PKD on two major benchmarks:

  • GLUE (General Language Understanding Evaluation): Seven tasks covering sentiment analysis, linguistic inference, and paraphrase detection.
  • RACE: A large-scale reading comprehension dataset derived from English exams for middle and high school students.

Key Results

Our experiments demonstrate that PKD significantly outperforms vanilla KD and standard fine-tuning (FT) across various model sizes.

Comparison of compressed BERT models on GLUE benchmark tasks.
  • Performance Retention: A 6-layer student model trained with PKD-Skip retains approximately 98-99% of the 12-layer teacher's performance. For instance, on the SST-2 sentiment task, the 6-layer PKD model achieved 92.0% accuracy compared to the teacher's 94.3%.
  • Efficiency Gains: The 6-layer PKD model offers a 1.94x speedup in inference and a 1.64x reduction in parameters. While 3-layer models provide even greater speedups (3.73x), they experience a steeper decline in accuracy.
  • Strategy Comparison: PKD-Skip generally outperformed PKD-Last, suggesting that the foundational knowledge in the teacher's lower layers is vital for the student's success.

Discussion and Insights

Learning from intermediate layers acts as a powerful regularizer, preventing the student from simply memorizing final outputs. We observed that PKD is particularly effective on larger datasets like MNLI and QNLI, where the student has sufficient data to capture complex internal representations. Furthermore, we found that normalizing hidden states was essential for stable training when calculating the PT loss.

Conclusion

Patient Knowledge Distillation provides a robust framework for compressing large language models without sacrificing significant accuracy. By focusing on the "journey" of the teacher's internal layers rather than just the final "destination," PKD enables the creation of efficient models suitable for resource-constrained environments. Future work will explore applying PKD during the pre-training phase and investigating alternative distance metrics for hidden state alignment.

Summary updated

Abstract

Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: ($i$) PKD-Last: learning from the last $k$ layers; and ($ii$) PKD-Skip: learning from every $k$ layers. These two patient distillation schemes enable the exploitation of rich information in the teacher's hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multi-layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training efficiency, without sacrificing model accuracy.
BibTeX
@inproceedings{sun2019patient,
  author = {Siqi Sun and Yu Cheng and Zhe Gan and Jingjing Liu},
  title = {{Patient Knowledge Distillation for BERT Model Compression}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=A63ESvpdmh7},
}
Figure