Abstract
We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers. In particular, we define multi-head self-attention relations as scaled dot-product between the pairs of query, key, and value vectors within each self-attention module. Then we employ the above relational knowledge to train the student model. Besides its simplicity and unified principle, more favorably, there is no restriction in terms of the number of student's attention heads, while most previous work has to guarantee the same head number between teacher and student. Moreover, the fine-grained self-attention relations tend to fully exploit the interaction knowledge learned by Transformer. In addition, we thoroughly examine the layer selection strategy for teacher models, rather than just relying on the last layer as in MiniLM. We conduct extensive experiments on compressing both monolingual and multilingual pretrained models. Experimental results demonstrate that our models distilled from base-size and large-size teachers (BERT, RoBERTa and XLM-R) outperform the state-of-the-art.
Introduction
Pretrained Transformers like BERT and RoBERTa have revolutionized NLP, but their massive size (hundreds of millions of parameters) makes them slow and expensive to deploy in real-world, low-latency applications. To solve this, researchers use Knowledge Distillation (KD), where a small "student" model is trained to behave like a large "teacher."
However, existing KD methods often suffer from two main issues:
- Architectural Rigidity: Many methods require the student to have the same number of attention heads as the teacher to "map" the knowledge correctly.
- Suboptimal Layer Selection: Most methods either map layers one-to-one or only use the teacher's very last layer. For extremely deep models (like the 24-layer BERT-Large), the final layer might not actually be the best source of generalizable knowledge for a small student.
Methods
The core innovation is Multi-Head Self-Attention Relation Distillation. Standard attention distillation usually focuses on the "attention map" (the relationship between Queries and Keys). MiniLMv2 goes deeper by distilling three types of relations: Query-Query (Q-Q), Key-Key (K-K), and Value-Value (V-V).
To make this work regardless of the number of heads, the authors implement Head Alignment: they concatenate the attention vectors from all heads and then re-split them into a new number of "relation heads." This allows a student with 12 heads to learn from a teacher with 16 heads seamlessly. The student is trained to minimize the difference (KL-Divergence) between its internal relation distributions and those of the teacher.
Strategic Layer Selection
Instead of blindly picking the last layer, the authors systematically tested which teacher layers provide the best "signal." One of the most surprising findings is that for 24-layer models, the last layer (layer 24) is not always the best teacher.
As shown in the experimental results, layers like 21 (for BERT-Large) or 19 (for XLM-R) often produced better students than the final layer. This suggests that the very top layers might become too specialized or "noisy" for a small model to digest effectively.
Experimental Results
MiniLMv2 was evaluated on massive standard corpora including Wikipedia, BookCorpus, and CC-100. Benchmarks included GLUE, SQuAD 2.0, XNLI, and MLQA. Key findings include:
- Efficiency: A 6-layer student distilled from BERT-Large is 2.0x faster than the standard BERT-Base but achieves higher accuracy on GLUE.
- Multilingual Performance: The 6-layer multilingual model outperformed the standard 12-layer mBERT while being 5.3x faster.
- Parameter Efficiency: A tiny 22M parameter model distilled with this method retained over 96% of the performance of a teacher nearly 15x its size.
Conclusion
MiniLMv2 demonstrates that mimicking raw attention weights is insufficient; capturing fine-grained relational understanding (Q-Q, K-K, V-V) is key to effective compression. By focusing only on self-attention relations and ignoring hidden state distillation, the training process is simplified without sacrificing accuracy. Future work includes developing automatic layer selection algorithms to replace the current empirical testing of teacher layers.
Summary updated