Abstract
Transformers have gradually become a key component for many state-of-the-art natural language representation models. A recent Transformer based model—BERT—achieved state-of-the-art results on various natural language processing tasks, including GLUE, SQuAD v1.1, and SQuAD v2.0. This model, however, is computationally prohibitive and has a huge number of parameters. In this work, we revisit the architecture choices of BERT in efforts to obtain a lighter model. We focus on reducing the number of parameters, yet our methods can be applied towards other objectives such as FLOPs or latency. We show that much more efficient light BERT models can be obtained by reducing algorithmically chosen architecture design dimensions rather than simply reducing the number of Transformer encoder layers. In particular, our schuBERT gives 6.6% higher average accuracy on GLUE and SQuAD datasets as compared to BERT with three encoder layers while having the same number of parameters.
Rethinking the Architecture of Efficiency
Large Language Models like BERT are notoriously heavy, often containing hundreds of millions of parameters. This makes them expensive to train and slow to deploy in production environments. Traditionally, attempts to slim down BERT followed two paths: depth reduction (using fewer layers) or parameter sharing (reusing weights).
The authors of schuBERT argue these approaches are suboptimal. They suggest that the "default" dimensions of the Transformer architecture—such as the fixed ratio between hidden size and feed-forward networks—are often arbitrary. The real challenge lies in finding the optimal "shape" for the model by deciding exactly how many attention heads, how much memory, and how much processing power each specific layer actually needs.
Methodology: Pruning-based Architecture Search
To find the optimal configuration without training thousands of models from scratch, the researchers introduced a granular way to parameterize BERT. While standard BERT-base uses fixed ratios across all 12 layers, schuBERT allows five dimensions to vary per layer: the number of attention heads ($a$), hidden size ($h$), feed-forward network (FFN) filter size ($f$), and the key-query ($k$) and value ($v$) dimensions.
The search utilizes a Pruning-based Architecture Search. Starting with a full BERT-base model, the team introduced "prunable" vectors for each dimension. By training the model with a sparsity-inducing penalty, the optimization process "squeezes" less important dimensions toward zero. Once identified, these unimportant parts are pruned away, leaving a specialized, "skinny" version of BERT that is then fine-tuned for maximum performance.
Key Insights: The "Tall and Narrow" Advantage
A striking finding of the search is that a 12-layer model with very "thin" layers performs significantly better than a 3-layer model with "thick" layers, even when they share the same parameter count.
As shown in the figure above, the FFN filter size ($f$) tends to increase as you move from the bottom to the top of the stack. This suggests that the top layers need more "brain power" to process complex representations. Furthermore, the classic Transformer design (where $f = 4h$) was found to be suboptimal for efficient models; schuBERT indicates that the FFN often needs to be much larger relative to the hidden state to maintain high accuracy.
Performance and Results
The authors compared schuBERT against standard BERT models scaled down by reducing layers (BERT-$\ell$). The results demonstrate that algorithmic dimension reduction is far superior to simple layer removal.
Key performance highlights include:
- Massive Accuracy Gains: At the 43M parameter mark, schuBERT achieved an average accuracy of 81.7%, while a standard 3-layer BERT with the same parameter count only managed 75.1%.
- Minimal Loss at Scale: A 99M parameter schuBERT nearly matched the original BERT-base (84.1% vs 84.3% average accuracy) while being more efficient.
- True Efficiency: Because schuBERT reduces the actual dimensions of the matrices, it provides a genuine reduction in computational latency rather than just a smaller file size.
Conclusion and Future Work
schuBERT demonstrates that the internal architecture of Transformers is ripe for optimization. While the current work focuses on parameter reduction, future iterations could apply these pruning methods directly to hardware-specific metrics like FLOPs or measured latency on mobile devices. Combining schuBERT’s optimized architecture with techniques like Knowledge Distillation could likely push the boundaries of model efficiency even further.
Summary updated