DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference

Cristobal Eyzaguirre, Felipe del Rio, Vladimir Araujo, Alvaro Soto

TL;DR

This paper introduces DACT-BERT, a novel approach to make large language models like BERT more efficient by allowing them to adaptively decide how many processing layers (Transformer blocks) are needed for each input. By learning to halt computation early when an answer is sufficiently confident, DACT-BERT achieves competitive performance with significant reductions in computational cost, outperforming existing dynamic methods.

What Problem Does This Paper Address?

Large pre-trained language models (LLMs) like BERT have revolutionized Natural Language Processing (NLP), achieving state-of-the-art results across many tasks. However, their immense size and complexity come at a significant computational cost, leading to long inference times and high energy consumption. This makes them impractical for low-end devices and raises environmental concerns. While many LLMs are over-parameterized, meaning they have redundant capacity, current models process every input with the same fixed number of layers, regardless of the input's complexity. This paper addresses the need for more efficient LLMs by developing a method that allows models to dynamically adjust their computation based on the input, thereby saving resources without sacrificing accuracy.

Methods

DACT-BERT builds upon the concept of Adaptive Computation Time (ACT) and integrates it into the BERT architecture. The core innovation is the addition of a "halting neuron" after each Transformer block. This neuron, trained to be differentiable, outputs a confidence score (halting value) indicating how certain the model is about its current prediction.

Here's a breakdown of the key methodological aspects:

  • Adaptive Computation Time (ACT): DACT-BERT uses a differentiable ACT mechanism. This allows the model to learn when to stop processing.
  • Halting Neuron: After each Transformer block, a new layer is added that outputs a confidence score. This score is crucial for deciding whether to continue processing.
  • Differentiable Training: The entire DACT-BERT model, including the halting mechanism, is end-to-end differentiable. This is a significant advantage, as it allows the weights of the underlying Transformer backbone to be fine-tuned jointly with the halting classifiers. This enables the model to learn not just when to stop, but also how to adapt its core processing for efficiency.
  • Accumulated Answer: The model maintains an "accumulated answer" that is updated at each layer. This answer is a combination of the current layer's output and the accumulated answer from previous layers, weighted by the halting probabilities.
  • Inference Halting Condition: During inference, the model halts when the confidence in the top-predicted class is sufficiently high compared to the runner-up classes, indicating that further computation is unlikely to change the final prediction. This condition is formulated to ensure that the decision to halt is robust.

Datasets

The paper evaluates DACT-BERT on six tasks from the GLUE benchmark (Wang et al., 2018). These tasks are:

  • BERT-QQP
  • BERT-SST-2
  • BERT-MNLI
  • BERT-QNLI
  • BERT-MRPC
  • BERT-RTE

The authors use both BERT-base and RoBERTa-base as backbones for their experiments.

Key Insights and Ideas

  • Learning to Halt is Key: The most significant insight is that by making the halting mechanism differentiable and jointly training it with the backbone, the model can learn to adapt its computation more effectively than methods relying on fixed heuristics.
  • Input-Dependent Computation: DACT-BERT demonstrates that not all inputs require the full computational depth of a Transformer model. The model can learn to "prune" layers for simpler inputs, leading to substantial efficiency gains.
  • Robustness to Hyperparameters: The paper highlights that DACT-BERT shows remarkable stability in the amount of computation saved across different values of its main hyperparameter (τ), unlike some other dynamic methods that are highly sensitive to their chosen parameters. This suggests a more reliable and predictable efficiency mechanism.
  • Full Backbone Fine-tuning: The ability to fine-tune the entire backbone is a critical differentiator. This allows the model to optimize its internal representations for both accuracy and efficient early exiting, a capability not present in all dynamic Transformer approaches.

Key Results

  • Competitive Performance: DACT-BERT achieves performance comparable to static BERT and RoBERTa baselines, and in many cases, it outperforms existing dynamic Transformer methods like DeeBERT and PaBEE, especially on tasks with more training data.
  • Significant Computation Reduction: The paper presents performance vs. efficiency trade-off curves (Figure 2) showing that DACT-BERT can achieve high accuracy while using a fraction of the Transformer layers. For instance, on some tasks, it uses significantly fewer layers than static models while maintaining accuracy.
  • Efficiency-Performance Diagrams: The efficiency-performance diagrams illustrate that DACT-BERT excels in the reduced computational regime, outperforming alternatives when using limited computation.
  • Frequency of Layer Usage: Figure 3 shows the frequency with which each Transformer block is used. This visualization provides insight into which layers are most critical and which can be more readily skipped.

Limitations and Future Work

  • Limitations: The paper acknowledges that while DACT-BERT outperforms existing dynamic methods, it is not explicitly aiming for state-of-the-art accuracy on all tasks. The focus is on the efficiency gains achieved through adaptive computation. The authors also note that the model's halting behavior can be influenced by the choice of the hyperparameter τ, which moderates the trade-off between complexity and error.
  • Future Work: The paper suggests that future work could explore further optimizations and applications of DACT-BERT. The authors mention that their model uses fewer layers compared to DeeBERT, allowing for pruning of final layers. This suggests that further investigation into layer pruning and its impact on performance could be a fruitful direction. The inherent differentiability of the approach also opens doors for its application in other complex neural architectures.

Summary updated

Abstract

Large-scale pre-trained language models have shown remarkable results in diverse NLP applications. Unfortunately, these performance gains have been accompanied by a significant increase in computation time and model size, stressing the need to develop new or complementary strategies to increase the efficiency of these models. In this paper we propose DACT-BERT, a differentiable adaptive computation time strategy for BERT-like models. DACT-BERT adds an adaptive computational mechanism to BERT's regular processing pipeline, which controls the number of Transformer blocks that need to be executed at inference time. By doing this, the model learns to combine the most appropriate intermediate representations for the task at hand. Our experiments demonstrate that our approach, when compared to the baselines, excels on a reduced computational regime and is competitive in other less restrictive ones.
BibTeX
@inproceedings{eyzaguirre2022dact,
  author = {Cristobal Eyzaguirre and Felipe del Rio and Vladimir Araujo and Alvaro Soto},
  title = {{DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure