DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference

Cristóbal Eyzaguirre, Felipe del Río, Vladimir Araujo, Álvaro Soto

TL;DR

This paper introduces DACT-BERT, a novel method that makes large language models like BERT more efficient by allowing them to dynamically decide how many processing layers (Transformer blocks) are needed for a given input. DACT-BERT achieves this through a differentiable mechanism that learns when a prediction is stable, leading to significant computational savings without sacrificing accuracy, especially on tasks with more training data.

What Problem Does This Paper Address?

Large pre-trained language models (LLMs) such as BERT have revolutionized Natural Language Processing (NLP), achieving state-of-the-art results across a wide range of tasks. However, their impressive performance comes at a steep cost: massive model sizes and high computational demands for inference. This makes them impractical for deployment on resource-constrained devices and contributes to a significant environmental footprint. While existing methods have explored making these models more efficient, they often still process every input through a fixed number of layers, regardless of the input's complexity. This paper addresses the need for LLMs that can adapt their computation dynamically, using fewer resources for simpler inputs and more for complex ones, thereby optimizing the trade-off between performance and efficiency.

Methods

The core innovation of DACT-BERT is the integration of a Differentiable Adaptive Computation Time (DACT) mechanism into the standard BERT architecture. Unlike previous "dynamic Transformer" approaches that often rely on heuristic halting criteria or separate, non-differentiable classifiers, DACT-BERT introduces a fully differentiable "halting neuron" after each Transformer block.

Here's how it works:

  • Adaptive Computation: After each Transformer block, DACT-BERT outputs not only the intermediate representation but also a "halting score." This score, learned end-to-end, indicates the model's confidence that the current prediction is stable and won't change significantly with further processing.
  • Differentiable Training: The entire DACT-BERT model, including the halting mechanism and the underlying Transformer blocks, is trained jointly. This "full differentiability" is crucial because it allows the model to fine-tune the weights of the Transformer backbone itself, enabling it to better adapt to dynamic computation strategies.
  • Inference Halting: During inference, the halting scores are used to determine when to stop processing. The model halts when it predicts that further computation will not alter the final output, effectively skipping unnecessary layers for simpler inputs. This is achieved by learning to interpolate between the current step's answer and the previous one, guided by the confidence scores.

Datasets

The paper evaluates DACT-BERT on six different tasks from the GLUE benchmark (General Language Understanding Evaluation). These tasks include:

  • BERT-QQP
  • BERT-SST-2
  • BERT-MRPC
  • BERT-MNLI
  • BERT-QNLI
  • BERT-RTE

The models were evaluated using both BERT and RoBERTa backbones.

Key Insights and Ideas

  • Full Differentiability is Key: The paper strongly emphasizes that the end-to-end differentiability of DACT-BERT is a significant advantage. This allows the model to learn not just when to stop, but also how the Transformer blocks should behave to facilitate efficient early exiting. This contrasts with methods that freeze the backbone and only train auxiliary classifiers.
  • Learning to Regulate Itself: DACT-BERT learns its own efficiency mechanism rather than relying on arbitrary heuristics (like entropy thresholds or fixed patience values used by some baselines). This self-regulation leads to more stable and robust performance across different tasks and hyperparameter settings.
  • Advantage with More Data: The authors observe that DACT-BERT particularly excels in tasks with more available training data. This suggests that the model can better learn the complex interplay between computation and performance when it has sufficient examples to generalize from.
  • Compressibility: Figure 3 in the paper shows that DACT-BERT tends to use fewer layers on average compared to DeeBERT, suggesting it's inherently more "compressible" and amenable to pruning.

Key Results

  • Improved Efficiency-Performance Trade-off: Figure 2, which plots performance against efficiency (fraction of layers used), demonstrates that DACT-BERT consistently outperforms dynamic baselines like DeeBERT and PaBEE, especially at lower computational regimes (fewer layers used).
  • Competitive Performance: While excelling in reduced computational settings, DACT-BERT remains competitive with static baselines (non-adaptive BERT) in less restrictive computational regimes.
  • Robustness: The model exhibits remarkable stability in the amount of computation saved, unlike DeeBERT, which is shown to be sensitive to its entropy hyperparameter.

Limitations and Future Work

  • Limitations: The paper acknowledges that while DACT-BERT offers significant improvements, it's still an adaptation of existing architectures. The focus was on demonstrating the effectiveness of the DACT mechanism rather than achieving absolute state-of-the-art results on all tasks. The paper also notes that the GLUE benchmark tasks are relatively standard, and further evaluation on more diverse or complex datasets could be beneficial.
  • Future Work: The authors suggest that their approach could be extended to other Transformer-based models beyond BERT and RoBERTa. Further research could explore different halting criteria or more sophisticated ways to combine intermediate representations. The potential for applying DACT-BERT to tasks beyond classification, such as sequence generation, is also an interesting avenue.

Summary updated

Abstract

Large-scale pre-trained language models have shown remarkable results in diverse NLP applications. Unfortunately, these performance gains have been accompanied by a significant increase in computation time and model size, stressing the need to develop new or complementary strategies to increase the efficiency of these models. In this paper we propose DACT-BERT, a differentiable adaptive computation time strategy for BERT-like models. DACT-BERT adds an adaptive computational mechanism to BERT's regular processing pipeline, which controls the number of Transformer blocks that need to be executed at inference time. By doing this, the model learns to combine the most appropriate intermediate representations for the task at hand. Our experiments demonstrate that our approach, when compared to the baselines, excels on a reduced computational regime and is competitive in other less restrictive ones.
BibTeX
@inproceedings{eyzaguirre2022dact,
  author = {Cristóbal Eyzaguirre and Felipe del Río and Vladimir Araujo and Álvaro Soto},
  title = {{DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure