Abstract: Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
While pre-trained Transformer models like BERT have set new benchmarks across natural language processing (NLP) tasks, their deployment remains challenging due to immense computational requirements. BERT-Base and BERT-Large contain 110 million and 330 million parameters, respectively, leading to significant inference latency and high resource costs.
Standard Knowledge Distillation (KD) typically involves a compact "student" model mimicking the final output (logits) of a larger "teacher" model. However, this approach often fails to capture the rich hierarchical linguistic features learned within the teacher's intermediate layers. This paper introduces Patient Knowledge Distillation (PKD), a framework designed to compress BERT by encouraging the student to "patiently" learn from the teacher's internal representations throughout the distillation process.
Methodology: The Patient Approach
The core innovation of PKD is the requirement that the student model aligns its hidden states with those of the teacher, rather than just matching the final prediction. This acts as a powerful regularizer, preventing the student from overfitting to the teacher's output distribution and helping it capture deeper linguistic structures.
Distillation Strategies
The authors explored two primary strategies for mapping teacher layers to student layers:
- PKD-Last: The student learns from the final k layers of the teacher, assuming that task-specific information is concentrated at the top of the network.
- PKD-Skip: The student learns from every k-th layer (e.g., layers 2, 4, 6, 8, and 10). This strategy assumes that a diverse range of features—from low-level syntax to high-level semantics—is necessary for a robust student model.
The Loss Function
The training objective combines three distinct components:
- Standard Cross-Entropy (CE Loss): Learning from ground-truth labels.
- Distillation Loss (DS Loss): Mimicking the teacher's soft probability outputs.
- Patient Teacher Loss (PT Loss): Minimizing the Mean Square Error (MSE) between the student's and teacher's hidden states, specifically focusing on the
[CLS]token representation.
Experimental Results
The authors evaluated PKD on the GLUE benchmark and the RACE reading comprehension dataset. They successfully compressed the 12-layer BERT-Base into 6-layer and 3-layer variants.
Performance and Efficiency
- Accuracy Retention: The 6-layer PKD model retained approximately 98%–99% of the teacher's performance on tasks like SST-2 and QNLI, significantly outperforming vanilla KD and direct fine-tuning.
- Inference Speedup: The 6-layer model achieved a 1.94x speedup, while the 3-layer model reached a 3.73x speedup in inference time.
- Generalization: As shown in the training curves, PKD models consistently outperform standard KD models, suggesting that internal guidance helps the student learn more robust features.
Key Insights
- Skip > Last: In most experiments, PKD-Skip outperformed PKD-Last. This indicates that capturing the evolution of representations across the teacher's entire depth is more beneficial than focusing solely on the final layers.
- Regularization Effect: Forcing internal alignment prevents the student from taking "shortcuts," leading to better generalization on test data compared to vanilla KD, which often plateaus early.
- Complexity Handling: On the reasoning-heavy RACE dataset, PKD provided a 1.6% performance boost over vanilla KD, proving its effectiveness for complex NLP tasks.
Conclusion and Future Work
Patient Knowledge Distillation provides a viable path for deploying high-performance BERT models in resource-constrained environments. While the method is sensitive to hyperparameters like the weight of the intermediate loss ($\beta$), it offers a significant leap in efficiency without sacrificing substantial accuracy. Future research directions include applying PKD during the pre-training phase and exploring more sophisticated distance metrics for hidden state alignment.
Summary updated