Abstract
Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved in two aspects: the relation-level knowledge could be further explored to improve model performance; and the setting of student attention head number could be more flexible to decrease inference time. Therefore, we are motivated to propose a novel knowledge distillation method MLKD-BERT to distill multi-level knowledge in teacher-student framework. Extensive experiments on GLUE benchmark and extractive question answering tasks demonstrate that our method outperforms state-of-the-art knowledge distillation methods on BERT. In addition, MLKD-BERT can flexibly set student attention head number, allowing for substantial inference time decrease with little performance drop.
Introduction
Large-scale Pre-trained Language Models (PLMs) like BERT have revolutionized NLP, but their massive size makes them difficult to deploy on resource-constrained devices. While knowledge distillation—training a small "student" model to mimic a large "teacher"—is a popular solution, it often comes with architectural strings attached.
Existing distillation methods for BERT (like TinyBERT or MiniLM) generally face two hurdles:
- Underutilized Relationships: Most methods focus on "feature-level" knowledge (matching hidden states), often ignoring "relation-level" knowledge—how tokens relate within a sentence or how different samples in a batch relate to one another.
- Architectural Rigidity: Many methods distill the self-attention distribution, forcing the student to have the exact same number of attention heads as the teacher.
MLKD-BERT addresses these issues by introducing a multi-level distillation approach that captures complex relationships while allowing for flexible student architectures.
Methodology
The MLKD-BERT framework utilizes a two-stage distillation process involving six distinct loss functions to transfer knowledge from a teacher to a student.
Stage 1: Internal Layer Distillation
In the first stage, the student learns from the teacher's internal representations:
- Embedding Layer: The student learns the token similarity relation (how word A relates to word B) rather than just matching raw vectors.
- Transformer Layers: This includes standard Feed-Forward Network (FFN) distillation and a novel MHA-split Distillation. By splitting concatenated Multi-Head Attention (MHA) outputs into groups and matching relations between these splits, the student’s head count is decoupled from the teacher’s.
Stage 2: Prediction Layer Distillation
The second stage focuses on the final output logic:
- Soft Labels: Matching the teacher’s final probability distribution.
- Sample Similarity & Contrastive Relation: The student learns how the teacher groups similar samples in a batch, using contrastive loss to ensure representations of samples with the same label are closer together.
Experimental Results
MLKD-BERT was evaluated on the GLUE benchmark and SQuAD tasks. The results demonstrate that MLKD-BERT consistently outperforms existing state-of-the-art distillation methods.
Key findings include:
- Superior Performance: MLKD-BERT$_4$ (a 4-layer student) outperformed models like TinyBERT and DistilBERT across most GLUE tasks.
- Efficiency: MLKD-BERT$_6$ achieved 99.5% of the performance of the full BERT-base teacher while being 2x faster and having 50% fewer parameters.
- Flexible Speedups: By reducing student attention heads from 12 to 3, researchers achieved an additional 14% decrease in inference time on MNLI with only a 1.7% accuracy drop.
- Robustness: On SQuAD question-answering tasks, MLKD-BERT consistently surpassed SOTA baselines.
Conclusion
MLKD-BERT provides a flexible and powerful framework for BERT compression. By focusing on multi-level relations and breaking architectural constraints, it enables the creation of highly efficient student models that retain the vast majority of their teacher's performance. While the two-stage process adds training overhead, the resulting gains in inference speed and accuracy make it a compelling choice for real-world deployment.
Summary updated