RefBERT: Compressing BERT by Referencing to Pre-computed Representations

Xinyi Wang, Haiqin Yang, Liang Zhao, Yang Mo, Jianping ShenIJCNN 2021 · OpenReview

TL;DR

This paper introduces RefBERT, a novel knowledge distillation method that compresses large BERT models into smaller, faster versions. Unlike previous methods, RefBERT leverages pre-computed teacher representations of similar "reference" samples during student training, theoretically increasing information absorption and empirically achieving competitive performance with significantly reduced size and latency.

What Problem Does This Paper Address?

Large pre-trained language models like BERT have revolutionized NLP, but their massive size and computational demands make them impractical for many real-world applications. Knowledge distillation is a common technique to create smaller "student" models that mimic the performance of larger "teacher" models. However, existing distillation methods often discard the teacher's valuable representations after training, potentially leading to suboptimal student performance. This paper aims to address this by proposing a method that continuously utilizes teacher knowledge during the compression process.

Methods

RefBERT builds upon the knowledge distillation framework, specifically inspired by TinyBERT. The core innovation lies in how it utilizes "reference samples." For each input sample, RefBERT finds a similar, but not identical, sample from the dataset. During the distillation process, the student model's first Transformer layer is modified to incorporate information from both the current input and its corresponding reference sample.

Specifically, the query, key, and value computations in the multi-head attention mechanism are augmented:

  • Key (K): Concatenates the student's input embedding with the teacher's embedding of the reference sample.
  • Value (V): Concatenates the student's input embedding with the teacher's final layer representation of the reference sample.

This allows the student to "see" how the teacher processes similar but distinct inputs. Additionally, RefBERT introduces a modification to the attention scores by subtracting a constant, which helps to amplify the effect of informative tokens and suppress non-informative ones. The training objective is a combination of MSE losses for distilling embeddings, hidden states, and attention weights, along with a cross-entropy loss for the prediction layer, similar to TinyBERT.

The authors also provide theoretical justifications for their approach, demonstrating that using MSE loss is a valid surrogate for information absorption and that incorporating reference samples increases the mutual information between the student and teacher models.

Datasets

  • General Distillation (GD) Stage:
    • English Wikipedia (2,500M words): Used as the unlabeled language model (DLM) dataset for pre-training the student model.
  • Reference Dataset: A new dataset is constructed by pairing each sample in the DLM with its most similar sample (using TF-IDF similarity via Elasticsearch) from the DLM itself.
  • Task-Specific Distillation (TD) Stage:
    • GLUE Benchmark: Used for evaluating the performance of the distilled models on various downstream NLP tasks. The paper lists 9 tasks within GLUE, including CoLA, MNLI, MRPC, QNLI, QQP, RTE, and STS-B.

Key Insights and Ideas

  • Leveraging Reference Samples: The central idea is that by exposing the student model to how the teacher processes similar inputs (via reference samples), the student can learn more effectively and retain more of the teacher's knowledge, even when the reference sample is not identical to the input.
  • Theoretical Justification for MSE: The paper provides a theoretical link between minimizing Mean Squared Error (MSE) loss and maximizing information absorption, justifying its use in the distillation process.
  • Mutual Information Boost: The theoretical analysis suggests that including reference samples in the distillation process can increase the mutual information between the student and teacher representations, leading to better knowledge transfer.
  • Attention Score Manipulation: Subtracting a constant from attention scores is a clever trick to make the attention mechanism more sensitive to relevant tokens and less influenced by noise.

Key Results

  • Performance: RefBERT achieves competitive performance on the GLUE benchmark, reaching over 94% of the BERTBASE performance on average.
  • Outperforming Baselines: RefBERT significantly outperforms the vanilla TinyBERT by 8.1% on average across four tasks. It also shows improvements over DistilBERT.
  • Efficiency: RefBERT is substantially more efficient than BERTBASE:
  • Size: 7.4x smaller.
  • Inference Speed: 9.5x faster.
  • Model Size vs. TinyBERT: RefBERT has a slightly larger model size than TinyBERT but achieves better performance.

Limitations and Future Work

  • Model Size Reduction: The authors suggest exploring further methods to reduce the model size of RefBERT while maintaining performance.
  • Wider/Deeper Teachers: Investigating how to transfer knowledge from even larger or more complex teacher models (e.g., BERT-large) using the reference mechanism is a promising direction.
  • Other Speed-up Methods: The paper mentions exploring other techniques like quantization, pruning, and hardware acceleration to further address the computational overhead of large language models.
  • QQP Performance: RefBERT showed relatively lower performance on the QQP task. The authors hypothesize that this might be due to reference samples being too similar to the evaluated samples, causing confusion. Further investigation into this specific task's behavior is warranted.

Summary updated

Abstract

Recently developed large pre-trained language models, e.g., BERT, have achieved remarkable performance in many downstream natural language processing applications. These pre-trained language models often contain hundreds of millions of parameters and suffer from high computation and latency in real-world applications. It is desirable to reduce the computation overhead of the models for fast training and inference while keeping the model performance in downstream applications. Several lines of work utilize knowledge distillation to compress the teacher model to a smaller student model. However, they usually discard the teacher's knowledge when in inference. Differently, in this paper, we propose RefBERT to leverage the knowledge learned from the teacher, i.e., facilitating the pre-computed BERT representation on the reference sample and compressing BERT into a smaller student model. To guarantee our proposal, we provide theoretical justification on the loss function and the usage of reference samples. Significantly, the theoretical result shows that including the pre-computed teacher's representations on the reference samples indeed increases the mutual information in learning the student model. Finally, we conduct the empirical evaluation and show that our RefBERT can beat the vanilla TinyBERT over 8.1\% and achieves more than 94\% of the performance of $\BERTBASE$ on the GLUE benchmark. Meanwhile, RefBERT is 7.4x smaller and 9.5x faster on inference than BERT$_{\rm BASE}$.
BibTeX
@inproceedings{wang2021refbert,
  author = {Xinyi Wang and Haiqin Yang and Liang Zhao and Yang Mo and Jianping Shen},
  title = {{RefBERT: Compressing BERT by Referencing to Pre-computed Representations}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=c-YOSTJNri},
}
Figure