Efficient Transformer Knowledge Distillation: A Performance Review

Nathan Brown, Ashton Williamson, Tahj Anderson, Logan LawrenceEMNLP 2023

TL;DR

This paper investigates the intersection of knowledge distillation (KD) and efficient transformer architectures for Natural Language Processing (NLP). The authors demonstrate that distilling knowledge from large, efficient transformer models into smaller ones can significantly reduce computational costs (inference time and memory) while retaining a high percentage of performance, making these models more accessible for real-world applications.

What Problem Does This Paper Address?

The NLP community has seen a surge in the performance of large transformer models like BERT. However, these models are computationally expensive and have limitations on input sequence length. To address this, "efficient attention" transformer architectures (e.g., Longformer, Big Bird) were developed to handle longer sequences with reduced overhead. Despite these advances, efficient transformers are still costly to train and deploy. Simultaneously, knowledge distillation has proven effective for model compression. This paper bridges these two areas by exploring how KD can be applied to efficient attention transformers to achieve both high performance and computational efficiency, particularly for long-context tasks. A key challenge addressed is the lack of a well-established long-context benchmark for Named Entity Recognition (NER), which the authors help to fill.

Methods

The core methodology employed is Convert-Then-Distill. This involves:

  1. Converting a pre-trained efficient attention teacher model to use its efficient attention mechanism.
  2. Distilling the knowledge from this converted teacher model into a smaller student model.
  3. Fine-tuning the student model on downstream tasks.

The authors specifically focus on the Convert-Then-Distill approach, as opposed to Distill-Then-Convert, believing it to be more practical for achieving efficient attention in student models. For distillation, they use a standard approach similar to DistilBERT, involving a combination of masked language modeling (MLM) loss, cross-entropy distillation loss, and cosine embedding loss. The student models' hidden layers are typically halved compared to the teacher.

Datasets

  • Training/Distillation:
    • OSCAR (Open Super-large Crawled Aggregated coRpus): A large, open-source corpus of raw, unannotated web text.
  • BookCorpus: A commonly used dataset for language model pretraining.
  • English Wikipedia: Used in conjunction with other datasets.
  • Evaluation:
    • GLUE: A benchmark suite for evaluating natural language understanding models on various tasks.
  • SQuAD (Stanford Question Answering Dataset): A dataset for question answering.
  • HotpotQA: A dataset for question answering requiring multi-hop reasoning.
  • TriviaQA: Another dataset for question answering.
  • CoNLL-2003: A standard benchmark for Named Entity Recognition (NER).
  • GONERD (Giant Oak NER Dataset): A newly introduced dataset by the authors for long-context NER. It is derived from web-scraped news and legal documents, featuring significantly longer sequences than CoNLL-2003, with approximately 35% of its sequences exceeding the 512-token threshold.

Key Insights and Ideas

  • Convert-Then-Distill is Effective: The paper strongly supports the Convert-Then-Distill paradigm as a viable strategy for creating performant and efficient transformer models, especially for long-context tasks.
  • Long-Context NER Gap Filled: The introduction of GONERD addresses a critical need for a benchmark that specifically evaluates NER models on longer sequences, which are increasingly relevant in real-world applications.
  • Efficiency Gains are Substantial: Distilled efficient models offer significant reductions in inference time (up to 57.8%) and memory usage, making them much more practical for deployment.
  • Performance Retention is High: Despite compression, distilled efficient models manage to retain a remarkably high percentage of their teacher's performance across various tasks, often exceeding 90%.
  • Distillation Data Matters: The choice of distillation dataset (OSCAR, BookCorpus, Wikipedia) has a noticeable impact on performance, with OSCAR + BookCorpus showing the best results.

Key Results

  • Performance Retention:
    • Short-context tasks (GLUE, SQuAD, CoNLL-2003): Up to 98.6% of original model performance.
  • Long-context QA (HotpotQA, TriviaQA): Up to 94.6% of original model performance.
  • Long-context NER (GONERD): Up to 98.8% of original model performance.
  • Inference Speedup: Up to 57.8% decrease in inference times.
  • Distilled Longformer-RoBERTa: Showed particularly strong results, retaining 95.9% of performance on GONERD for 50.7% of the cost.
  • Distilled Nyströmformer: Retained the highest average performance across QA benchmarks (94.7%).
  • Distilled Longformer: Achieved the fastest inference speed and lowest peak GPU memory usage among distilled efficient students, despite having more parameters.
  • Extending Sequence Length: Allowing standard BERT models to process longer sequences (4096 tokens) incurs significant speed and memory penalties, highlighting the necessity of efficient attention architectures.

Limitations and Future Work

  • Generalization of Efficient Students: The paper notes that Convert-Then-Distill can degrade performance more than separate Convert or KD steps, suggesting room for improvement in the generalization capabilities of efficient students.
  • Tailoring Distillation: The authors suggest that developing distillation methods specifically tailored to individual efficient attention mechanisms, tasks, and architectures could further reduce the student-teacher performance gap.
  • Optimal Distillation Paradigm: The Convert-Then-Distill paradigm is explored, but the paper acknowledges that other approaches (like Distill-Then-Convert) might also be fruitful and warrant further investigation.
  • GONERD Domain Bias: The GONERD dataset has a domain bias towards news and legal text, which might require additional pretraining for broader NER applications.
  • Distillation Data Mixture: For efficient attention models, distilling using a mixture of short and long-context data appears crucial for maximizing student performance, as indicated by ablation studies.

Summary updated

BibTeX
@inproceedings{brown2023efficient,
  author = {Nathan Brown and Ashton Williamson and Tahj Anderson and Logan Lawrence},
  title = {{Efficient Transformer Knowledge Distillation: A Performance Review}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure