PALBERT: Teaching ALBERT to Ponder

Nikita Balagansky, Daniil Gavrilov

TL;DR

This paper introduces PALBERT, an improved version of PonderNet for adaptive computation time in NLP models like ALBERT and RoBERTa. By replacing PonderNet's probabilistic early exit with a deterministic "Q-exit" criterion and enhancing the architecture, PALBERT achieves better accuracy and efficiency on downstream tasks, addressing the "overthinking" problem in large language models.

What Problem Does This Paper Address?

Large pre-trained language models (like BERT, ALBERT, RoBERTa) achieve state-of-the-art results but come with a significant computational cost. A key challenge is that these models often process every input through all their layers, even when a decision could be made earlier. This "overthinking" can lead to overconfidence in incorrect predictions and unnecessary computation. Adaptive Computation Time (ACT) methods, like PonderNet, aim to solve this by allowing models to exit early. However, PonderNet's original sampling-based exit criterion introduces randomness and variance, negatively impacting performance. This paper tackles the problem of making early exiting more robust and effective, particularly for ALBERT-like architectures.

Methods

The core contribution is the development of PALBERT, which builds upon PonderNet but introduces two key innovations:

  • Deterministic Q-exit Criterion: Instead of sampling an exit layer index from a learned probability distribution (which introduces variance), PALBERT uses a deterministic approach. It evaluates the Cumulative Distribution Function (CDF) of the exit probability at each layer. When the CDF exceeds a predefined threshold (q), the model exits. This "Q-exit" (Quantile-exit) ensures a consistent exit layer for a given input and threshold, reducing variance.
  • Revisited Lambda Layer Architecture: The original PonderNet used a simple MLP (Lambda layer) to predict the exit probability based on a single layer's hidden state. PALBERT enhances this by making the Lambda layer aware of the dynamics across layers. It achieves this by concatenating the current layer's hidden state with the previous layer's hidden state ($h_i$ and $h_{i-1}$) as input to the Lambda layer. Additionally, they explore using a 3-layer MLP for the Lambda layer and fine-tuning it with a separate learning rate.

Datasets

The models were evaluated on the GLUE Benchmark datasets, a standard suite for evaluating NLP models on a variety of tasks. Specifically, they used:

  • SST-2 (Sentiment Analysis)
  • RTE (Recognizing Textual Entailment)
  • QNLI (Question Answering Natural Language Inference)
  • CoLA (Corpus of Linguistic Acceptability)
  • MRPC (Microsoft Research Paraphrase Corpus)
  • MNLI (Multi-Genre Natural Language Inference)
  • QQP (Quora Question Pairs)
  • STS-B (Semantic Textual Similarity Benchmark)

No new benchmarks were introduced.

Key Insights and Ideas

  • Randomness is Detrimental: The paper strongly suggests that the inherent randomness in PonderNet's sampling-based exit criterion is a significant bottleneck, leading to reduced performance compared to deterministic approaches.
  • Deterministic Exit is Key: The proposed Q-exit criterion, by providing a deterministic path, significantly improves robustness and performance, making early exiting more practical.
  • Context Matters for Exit Decisions: Making the Lambda layer aware of the temporal dynamics of hidden states across layers (by concatenating $h_i$ and $h_{i-1}$) is crucial for better exit decisions, suggesting that early exit is not just about the current state but also the trajectory.
  • Threshold as a Trade-off: The Q-exit threshold ($q$) acts as a direct knob to control the trade-off between inference speed and accuracy, allowing users to tune it based on their specific needs.
  • Auxiliary Tasks Improve Performance: Even when the Q-exit threshold forces an exit from the last layer (effectively behaving like a standard model), PALBERT still shows improved accuracy on some tasks, suggesting that the auxiliary Lambda layers themselves contribute positively.

Key Results

  • Improved Accuracy: PALBERT consistently outperformed the original PonderNet architecture and the PABEE baseline on a wide range of GLUE tasks.
  • Comparable to Plain Models: PALBERT achieved performance comparable to plain ALBERT fine-tuning, while offering significant speedups.
  • Speed-up: The adaptive nature of PALBERT allows for faster inference compared to models that process all layers. Figure 3 shows speed-up plots, indicating that PALBERT can achieve good performance with a fraction of the inference time of vanilla ALBERT.
  • Ablation Study Findings:
    • The deterministic Q-exit criterion significantly improved accuracy over the sampling criterion.
  • A more complex Lambda layer architecture (3-layer MLP, $h_i$ + $h_{i-1}$ input) further boosted performance.
  • Vanilla PonderNet performed worse than plain ALBERT.
  • PonderNet evaluated in expectation (a non-early-exit baseline) performed better than the sampling criterion, reinforcing the issue with randomness.
  • Q-exit Threshold: A threshold of $q=0.5$ was found to provide the best overall performance across tasks. Higher values of $q$ tended to force exits from later layers, reducing speed benefits.

Limitations and Future Work

  • Prior Distribution Dependency: The paper acknowledges that the performance of PonderNet (and by extension, PALBERT) is sensitive to the choice of the prior distribution on exit layer indices. Finding an appropriate prior is crucial but challenging.
  • Lack of Theoretical Justification for Q-exit Threshold: While $q=0.5$ worked well empirically, there's no strong theoretical explanation for why this specific value is optimal. Further research into deterministic exit criteria is needed.
  • Outperforming Plain Models: PALBERT did not consistently outperform plain ALBERT and RoBERTa on all GLUE tasks, indicating room for improvement.
  • Future Directions:
    • Developing better parameterizations for the prior distribution on exit layer indices.
  • Investigating alternative deterministic exit criteria beyond the CDF threshold.
  • Exploring the addition of more auxiliary tasks or self-distillation mechanisms to further enhance PALBERT's performance.
  • Moving away from variational inference and explicit prior knowledge for training-level early exiting.

Summary updated

Abstract

Currently, pre-trained models can be considered the default choice for a wide range of NLP tasks. Despite their SoTA results, there is practical evidence that these models may require a different number of computing layers for different input sequences, since evaluating all layers leads to overconfidence in wrong predictions (namely overthinking). This problem can potentially be solved by implementing adaptive computation time approaches, which were first designed to improve inference speed. Recently proposed PonderNet may be a promising solution for performing an early exit by treating the exit layer's index as a latent variable. However, the originally proposed exit criterion, relying on sampling from trained posterior distribution on the probability of exiting from the $i$-th layer, introduces major variance in exit layer indices, significantly reducing the resulting model's performance. In this paper, we propose improving PonderNet with a novel deterministic Q-exit criterion and a revisited model architecture. We adapted the proposed mechanism to ALBERT and RoBERTa and compared it with recent methods for performing an early exit. We observed that the proposed changes can be considered significant improvements on the original PonderNet architecture and outperform PABEE on a wide range of GLUE tasks. In addition, we also performed an in-depth ablation study of the proposed architecture to further understand Lambda layers and their performance.
BibTeX
@inproceedings{balagansky2022palbert,
  author = {Nikita Balagansky and Daniil Gavrilov},
  title = {{PALBERT: Teaching ALBERT to Ponder}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure