MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

Ying Zhang, Ziheng Yang, Shufan Ji

Abstract

Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved in two aspects: the relation-level knowledge could be further explored to improve model performance; and the setting of student attention head number could be more flexible to decrease inference time. Therefore, we are motivated to propose a novel knowledge distillation method MLKD-BERT to distill multi-level knowledge in teacher-student framework. Extensive experiments on GLUE benchmark and extractive question answering tasks demonstrate that our method outperforms state-of-the-art knowledge distillation methods on BERT. In addition, MLKD-BERT can flexibly set student attention head number, allowing for substantial inference time decrease with little performance drop.

Introduction

Large-scale Pre-trained Language Models (PLMs) like BERT have revolutionized NLP, but their massive size makes them difficult to deploy on resource-constrained devices. While knowledge distillation—training a small "student" model to mimic a large "teacher"—is a popular solution, it often comes with architectural strings attached.

Existing distillation methods for BERT (like TinyBERT or MiniLM) generally face two hurdles:

  1. Underutilized Relationships: Most methods focus on "feature-level" knowledge (matching hidden states), often ignoring "relation-level" knowledge—how tokens relate within a sentence or how different samples in a batch relate to one another.
  2. Architectural Rigidity: Many methods distill the self-attention distribution, forcing the student to have the exact same number of attention heads as the teacher.

MLKD-BERT addresses these issues by introducing a multi-level distillation approach that captures complex relationships while allowing for flexible student architectures.

Methodology

The MLKD-BERT framework utilizes a two-stage distillation process involving six distinct loss functions to transfer knowledge from a teacher to a student.

Overview of the MLKD-BERT two-stage distillation framework showing internal layer and prediction layer distillation.

Stage 1: Internal Layer Distillation

In the first stage, the student learns from the teacher's internal representations:

  • Embedding Layer: The student learns the token similarity relation (how word A relates to word B) rather than just matching raw vectors.
  • Transformer Layers: This includes standard Feed-Forward Network (FFN) distillation and a novel MHA-split Distillation. By splitting concatenated Multi-Head Attention (MHA) outputs into groups and matching relations between these splits, the student’s head count is decoupled from the teacher’s.

Stage 2: Prediction Layer Distillation

The second stage focuses on the final output logic:

  • Soft Labels: Matching the teacher’s final probability distribution.
  • Sample Similarity & Contrastive Relation: The student learns how the teacher groups similar samples in a batch, using contrastive loss to ensure representations of samples with the same label are closer together.

Experimental Results

MLKD-BERT was evaluated on the GLUE benchmark and SQuAD tasks. The results demonstrate that MLKD-BERT consistently outperforms existing state-of-the-art distillation methods.

Comparison of MLKD-BERT with other distilled BERT models on the GLUE benchmark.

Key findings include:

  • Superior Performance: MLKD-BERT$_4$ (a 4-layer student) outperformed models like TinyBERT and DistilBERT across most GLUE tasks.
  • Efficiency: MLKD-BERT$_6$ achieved 99.5% of the performance of the full BERT-base teacher while being 2x faster and having 50% fewer parameters.
  • Flexible Speedups: By reducing student attention heads from 12 to 3, researchers achieved an additional 14% decrease in inference time on MNLI with only a 1.7% accuracy drop.
  • Robustness: On SQuAD question-answering tasks, MLKD-BERT consistently surpassed SOTA baselines.

Conclusion

MLKD-BERT provides a flexible and powerful framework for BERT compression. By focusing on multi-level relations and breaking architectural constraints, it enables the creation of highly efficient student models that retain the vast majority of their teacher's performance. While the two-stage process adds training overhead, the resulting gains in inference speed and accuracy make it a compelling choice for real-world deployment.

Summary updated

Abstract

Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical model BERT, they could be further improved in two aspects: the relation-level knowledge could be further explored to improve model performance; and the setting of student attention head number could be more flexible to decrease inference time. Therefore, we are motivated to propose a novel knowledge distillation method MLKD-BERT to distill multi-level knowledge in teacher-student framework. Extensive experiments on GLUE benchmark and extractive question answering tasks demonstrate that our method outperforms state-of-the-art knowledge distillation methods on BERT. In addition, MLKD-BERT can flexibly set student attention head number, allowing for substantial inference time decrease with little performance drop.
BibTeX
@misc{zhang2024mlkd,
  author = {Ying Zhang and Ziheng Yang and Shufan Ji},
  title = {{MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models}},
  year = {2024},
  eprint = {2407.02775},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.02775},
}
Figure