MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei

TL;DR

MiniLMv2 is a powerful evolution in model compression that allows small "student" Transformers to mimic the complex internal relationships of massive "teacher" models. By focusing on the deep relations between queries, keys, and values within self-attention modules, the authors achieve state-of-the-art performance while removing the common restriction that students must have the same number of attention heads as their teachers.

What Problem Does This Paper Address?

Pretrained Transformers like BERT and RoBERTa have revolutionized NLP, but their massive size (hundreds of millions of parameters) makes them slow and expensive to deploy in real-world, low-latency applications.

To solve this, researchers use Knowledge Distillation (KD), where a small "student" model is trained to behave like a large "teacher." However, existing KD methods often suffer from two main issues:

  1. Architectural Rigidity: Many methods require the student to have the same number of attention heads as the teacher to "map" the knowledge correctly.
  2. Suboptimal Layer Selection: Most methods either map layers one-to-one or only use the teacher's very last layer. For extremely deep models (like the 24-layer BERT-Large), the final layer might not actually be the best source of generalizable knowledge for a small student.

Methods

The core innovation is Multi-Head Self-Attention Relation Distillation.

Standard attention distillation usually focuses on the "attention map" (the relationship between Queries and Keys). MiniLMv2 goes deeper by distilling three types of relations: Query-Query (Q-Q), Key-Key (K-K), and Value-Value (V-V).

To make this work regardless of the number of heads:

  • Head Alignment: The authors concatenate the attention vectors from all heads and then re-split them into a new number of "relation heads." This allows a student with 12 heads to learn from a teacher with 16 heads seamlessly.
  • KL-Divergence Loss: The student is trained to minimize the difference (Kullback–Leibler divergence) between its internal relation distributions and those of the teacher.
  • Strategic Layer Selection: Instead of blindly picking the last layer, they systematically tested which teacher layers provide the best "signal."

Datasets

The models were pretrained on massive, standard corpora:

  • English: Wikipedia and BookCorpus.
  • Multilingual: The CC-100 dataset (used for XLM-R distillation).

For evaluation, the authors used the industry-standard benchmarks:

  • GLUE: A collection of nine diverse natural language understanding tasks.
  • SQuAD 2.0: A rigorous extractive question-answering dataset.
  • XNLI & MLQA: Benchmarks for cross-lingual understanding and question answering.

Key Insights and Ideas

  • Relations > Distributions: Mimicking the raw attention weights isn't enough. By forcing the student to learn how queries relate to other queries (Q-Q) and keys to keys (K-K), the student captures a much finer-grained understanding of the teacher’s internal logic.
  • The "Upper-Middle" Sweet Spot: One of the most surprising findings is that for 24-layer models, the last layer (layer 24) is not the best teacher. The authors found that layer 21 (for BERT) or layer 19 (for RoBERTa) produced better students. It suggests that the very top layers might become too specialized or "noisy" for a small model to digest.
  • Efficiency via Simplification: By focusing only on self-attention relations and ignoring hidden state distillation (which requires complex linear projections to match dimensions), the training process is simplified without sacrificing accuracy.

Key Results

MiniLMv2 consistently outperformed previous heavyweights like DistilBERT, TinyBERT, and the original MiniLM.

  • Better than the Original: A 6-layer student distilled from BERT-Large is 2.0x faster than the standard BERT-Base but actually achieves higher accuracy on GLUE.
  • Multilingual Dominance: Their 6-layer multilingual model outperformed the standard mBERT (12 layers) while being 5.3x faster.
  • Parameter Efficiency: A tiny 22M parameter model (6 layers, 384 hidden size) distilled with this method retained over 96% of the performance of a teacher nearly 15x its size.

Limitations and Future Work

While MiniLMv2 is highly effective, the authors noted a few areas for growth:

  • Manual Layer Selection: Currently, finding the "best" teacher layer requires empirical testing (e.g., trying layers 19, 20, and 21). They suggest that an automatic layer selection algorithm would be a valuable next step.
  • Task-Specific Nuance: While the model excels at general tasks, it performed slightly worse on CoLA (linguistic acceptability) compared to models that use specific training objectives like Masked Language Modeling (MLM).
  • Scaling Up: The authors are interested in seeing how this relational distillation scales when compressing even larger "mega-models" beyond the 340M parameter range.

Summary updated

Abstract

We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers. In particular, we define multi-head self-attention relations as scaled dot-product between the pairs of query, key, and value vectors within each self-attention module. Then we employ the above relational knowledge to train the student model. Besides its simplicity and unified principle, more favorably, there is no restriction in terms of the number of student's attention heads, while most previous work has to guarantee the same head number between teacher and student. Moreover, the fine-grained self-attention relations tend to fully exploit the interaction knowledge learned by Transformer. In addition, we thoroughly examine the layer selection strategy for teacher models, rather than just relying on the last layer as in MiniLM. We conduct extensive experiments on compressing both monolingual and multilingual pretrained models. Experimental results demonstrate that our models distilled from base-size and large-size teachers (BERT, RoBERTa and XLM-R) outperform the state-of-the-art.
BibTeX
@inproceedings{wang2021minilmv,
  author = {Wenhui Wang and Hangbo Bao and Shaohan Huang and Li Dong and Furu Wei},
  title = {{MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=il0t3c4laDZ},
}

Loading related content...

Figure