schuBERT: Optimizing Elements of BERT

Ashish Khetan, Zohar S. Karnin

TL;DR

This paper introduces "schuBERT," a method for optimizing BERT's architecture by fine-tuning individual layer dimensions (like attention heads, hidden size, and feed-forward network sizes) rather than just reducing the number of layers. This approach leads to significantly smaller and more efficient BERT models that can even outperform BERT with more layers on downstream tasks.

What Problem Does This Paper Address?

BERT, a powerful Transformer-based model, has achieved state-of-the-art results across many NLP tasks. However, its massive size and computational cost make it impractical for many real-world applications. While previous work has explored reducing BERT's size by decreasing the number of encoder layers, this often leads to a significant drop in performance. The authors aim to find a more effective way to create smaller, more efficient BERT models without sacrificing accuracy.

Methods

The core of this work is a novel optimization technique called "pruning-based architecture search." Instead of simply removing layers, the authors treat various architectural dimensions within each BERT encoder layer (number of attention heads, hidden size, feed-forward dimension, key/query/value dimensions) as tunable parameters. They introduce "prune-parameters" for each of these dimensions, initialized to ones. These parameters are then optimized using a combination of the original BERT loss functions (Masked Language Model and Next-Sentence Prediction) and a sparsity-inducing L1 regularization term. This process effectively "prunes" away less important dimensions by setting their corresponding prune-parameters to zero. After identifying the optimal pruned dimensions, the model is fine-tuned. This approach allows for a more granular optimization of the BERT architecture.

Datasets

  • Pre-training: BooksCorpus (800M words) and English Wikipedia (2,500M words).
  • Downstream Tasks (Evaluation):
    • Question Answering: SQuAD v1.1, SQuAD v2.0
    • GLUE Benchmark: MNLI, MRPC, SST-2, RTE

Key Insights and Ideas

  • Beyond Layer Reduction: Simply reducing the number of BERT layers is suboptimal. Optimizing individual architectural dimensions within layers offers a more promising path to efficiency.
  • "Tall and Narrow" is Better: For computationally lighter models, a "tall" architecture (more layers, smaller hidden dimensions) generally performs better than a "wide and shallow" one (fewer layers, larger hidden dimensions).
  • Top Layers Matter More: The fully-connected components in the upper layers of BERT play a more significant role than those in the lower layers.
  • Transformer Defaults are Suboptimal: The default architectural choices in the original Transformer paper (and subsequently BERT) for dimensions like key-query, value, and feed-forward filters might not be optimal for all scenarios.
  • Granular Optimization Pays Off: By allowing each encoder layer to have different design dimensions (e.g., varying feed-forward filter sizes across layers), significant performance gains can be achieved.

Key Results

  • schuBERT-all (99M parameters): Achieved 0.2% lower accuracy than BERT-base (108M parameters) on average across tasks, while being smaller.
  • schuBERT-h (43M parameters): Outperformed BERT-l (with 3 layers) by 6.6% higher average accuracy on GLUE and SQuAD datasets, despite having significantly fewer parameters. This highlights the effectiveness of optimizing hidden units and other dimensions over just reducing layers.
  • "Tall and Narrow" BERT: schuBERT models with 12 layers consistently outperformed BERT models with fewer layers (9, 6, 3) when the parameter count was matched.
  • Expansive Multi-head Attention: For smaller models, increasing key, query, and value dimensions (beyond the standard h/a) and feed-forward filter sizes (beyond 4h) led to better performance.

Limitations and Future Work

  • Complexity of Full Pruning: The authors note that while they expect schuBERT-all (pruning all dimensions) to perform best, its performance was sometimes slightly lower than schuBERT-h (pruning only hidden size), possibly due to the high complexity of jointly pruning all dimensions.
  • Complementary Techniques: The paper mentions that techniques like knowledge distillation (used in DistilBERT) are complementary and could be used to further boost schuBERT's performance.
  • Future Directions: The paper implicitly suggests further exploration into optimal architectural configurations for various model sizes and task types, and investigating the impact of these optimized dimensions on latency and FLOPs.

Summary updated

Abstract

Transformers \citep{vaswani2017attention} have gradually become a key component for many state-of-the-art natural language representation models. A recent Transformer based model- BERT \citep{devlin2018bert} achieved state-of-the-art results on various natural language processing tasks, including GLUE, SQuAD v1.1, and SQuAD v2.0. This model however is computationally prohibitive and has a huge number of parameters. In this work we revisit the architecture choices of BERT in efforts to obtain a lighter model. We focus on reducing the number of parameters yet our methods can be applied towards other objectives such FLOPs or latency. We show that much efficient light BERT models can be obtained by reducing algorithmically chosen correct architecture design dimensions rather than reducing the number of Transformer encoder layers. In particular, our schuBERT gives $6.6\%$ higher average accuracy on GLUE and SQuAD datasets as compared to BERT with three encoder layers while having the same number of parameters.
BibTeX
@inproceedings{khetan2020schubert,
  author = {Ashish Khetan and Zohar S. Karnin},
  title = {{schuBERT: Optimizing Elements of BERT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=0lJ_EUKAS_Fp},
}
Figure