PALBERT: Teaching ALBERT to Ponder

Nikita Balagansky, Daniil Gavrilov

TL;DR

PALBERT improves upon the PonderNet early-exit mechanism by replacing its stochastic sampling with a deterministic "Q-exit" criterion and a more robust architecture. By making the model "ponder" more effectively, the authors achieve better accuracy and faster inference on GLUE tasks compared to existing adaptive computation methods like PABEE.

What Problem Does This Paper Address?

Modern NLP models, like BERT and ALBERT, are often "over-parameterized" for many tasks. They typically process every input through a fixed number of layers, regardless of the input's complexity. This leads to two main issues:

  1. Computational Waste: Simple sentences don't need 12 layers of processing, which wastes energy and increases latency.
  2. Overthinking: Evaluating all layers can sometimes lead to "overthinking," where the model becomes overconfident in a wrong prediction as it processes the data further.

While "early exit" strategies (Adaptive Computation Time) exist, previous approaches like PonderNet relied on random sampling to decide when to stop. This randomness introduces high variance during inference—meaning the same input might exit at layer 2 one time and layer 10 the next—significantly hurting performance and reliability in production environments.

Methods

The authors propose two primary improvements to the PonderNet framework, specifically adapted for ALBERT and RoBERTa:

Comparison of stochastic sampling versus deterministic Q-exit.
  • Q-exit (Quantile Exit): This is a novel, zero-variance deterministic criterion. Instead of sampling from a probability distribution at each layer, the model calculates the Cumulative Distribution Function (CDF) of the exit probabilities. Once the CDF exceeds a specific threshold $q$, the model performs an early exit. This removes the "luck of the draw" from inference.
  • Revisited Lambda Layer: The "Lambda layer" is the component that predicts whether to exit. The authors upgraded this from a simple linear layer to a 3-layer MLP with a tanh activation function.
  • State Dynamics ($h$ concat): To help the model understand if it's still learning anything new, they feed the Lambda layer both the current hidden state ($h_i$) and the previous hidden state ($h_{i-1}$). This allows the model to sense the "dynamics" or changes between layers, making a more informed decision on whether further pondering is useful.

Datasets

The researchers evaluated their models on the GLUE (General Language Understanding Evaluation) benchmark, a standard suite for assessing NLU systems. Specifically, they reported results on SST-2 (Sentiment Analysis), RTE (Recognizing Textual Entailment), CoLA (Linguistic Acceptability), QNLI, MRPC, MNLI, QQP, and STS-B.

Key Insights and Ideas

  • Determinism is King: The shift from stochastic sampling to the deterministic Q-exit was the single most impactful change for stabilizing performance.
  • Contextual Stopping: Deciding to stop shouldn't just be based on what the model knows now, but how much that knowledge has changed since the last layer. Concatenating hidden states ($h_i, h_{i-1}$) provides this "velocity" of learning.
  • The 0.5 Threshold: Interestingly, a threshold of $q=0.5$ (the median of the distribution) consistently provided the best trade-off between speed and accuracy across various tasks.
Performance across different quantile thresholds on the MRPC dataset.
  • Better than Vanilla: In some cases (like SST-2), PALBERT actually performed better than a standard ALBERT model that uses all layers, suggesting that the auxiliary tasks at each layer act as a beneficial form of regularization.

Key Results

  • Superiority over PABEE: PALBERT outperformed PABEE (a popular consensus-based early exit method) across a wide range of GLUE tasks. For example, on the CoLA test set, PALBERT achieved a score of 48.6 compared to PABEE's 46.0.
  • Speed vs. Accuracy: PALBERT maintained performance comparable to full-layer ALBERT fine-tuning while significantly exceeding it in inference speed.
  • Macro Scores: On the ALBERT-Base backbone, PALBERT achieved a Macro score of 80.6 on the test set, outperforming both PABEE (79.3) and the original PonderNet implementation (77.2).

Limitations and Future Work

  • Prior Sensitivity: The model's performance is highly dependent on the "prior" distribution ($\lambda$). Finding the right $\lambda$ requires a more intensive hyperparameter search than simpler methods like PABEE.
  • Theoretical Gap: While $q=0.5$ works best empirically, the authors note there is currently no theoretical justification for why this specific threshold is optimal.
  • Future Directions: The authors suggest moving away from variational inference toward other trainable mechanisms that don't rely so heavily on explicit prior knowledge of exit layers. They also propose exploring self-distillation across layers to further boost accuracy.

Summary updated

Abstract

Currently, pre-trained models can be considered the default choice for a wide range of NLP tasks. Despite their SoTA results, there is practical evidence that these models may require a different number of computing layers for different input sequences, since evaluating all layers leads to overconfidence in wrong predictions (namely overthinking). This problem can potentially be solved by implementing adaptive computation time approaches, which were first designed to improve inference speed. Recently proposed PonderNet may be a promising solution for performing an early exit by treating the exit layer's index as a latent variable. However, the originally proposed exit criterion, relying on sampling from trained posterior distribution on the probability of exiting from the $i$-th layer, introduces major variance in exit layer indices, significantly reducing the resulting model's performance. In this paper, we propose improving PonderNet with a novel deterministic Q-exit criterion and a revisited model architecture. We adapted the proposed mechanism to ALBERT and RoBERTa and compared it with recent methods for performing an early exit. We observed that the proposed changes can be considered significant improvements on the original PonderNet architecture and outperform PABEE on a wide range of GLUE tasks. In addition, we also performed an in-depth ablation study of the proposed architecture to further understand Lambda layers and their performance.
BibTeX
@inproceedings{balagansky2022palbert,
  author = {Nikita Balagansky and Daniil Gavrilov},
  title = {{PALBERT: Teaching ALBERT to Ponder}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure