How Many Layers and Why? An Analysis of the Model Depth in Transformers

Antoine Simoulin, Benoît CrabbéACL 2021 · OpenReview

TL;DR

This paper investigates the role of multiple layers in Transformer models, proposing a variant of ALBERT that dynamically adjusts the number of layers applied to each token. They demonstrate that not all tokens require the same amount of processing, with more "difficult" or crucial tokens benefiting from more iterative refinement, offering a new perspective on Transformer layer functionality beyond simple feature extraction.

What Problem Does This Paper Address?

Transformers, particularly large language models like BERT, are known for their extensive parameterization, often involving many stacked layers. However, the precise function and necessity of these multiple layers remain somewhat unclear. While a common intuition is that lower layers capture surface information and higher layers capture semantics, this paper questions whether the number of layer applications is the key factor in abstracting information. Furthermore, fixed-depth models might be inefficient, processing simple tokens as extensively as complex ones. This research aims to understand the role of layer depth by exploring whether a dynamic, token-specific layer application strategy can be effective and what insights it provides into the model's internal workings.

Methods

The authors propose a modified ALBERT architecture. ALBERT is chosen because it ties weights across layers, meaning each layer applies the same transformation function. This allows the authors to conceptualize the application of a layer as an "iteration." Their key innovation is a "halting mechanism," adapted from Graves (2016), which dynamically decides when to stop processing a token. This mechanism assigns a probability to stop at each iteration and uses an update weight to combine the previous and current hidden states.

To encourage efficiency, they introduce a "ponder cost" into the pre-training objective, penalizing the total number of iterations across all tokens. This encourages the model to be parsimonious with its computational resources. The model is trained using the standard ALBERT pre-training tasks: Masked Language Model (MLM) and Sentence Order Prediction (SOP).

Datasets

  • BOOKCORPUS: Used for pre-training the models.
  • English Wikipedia: Also used for pre-training.
  • GLUE benchmark: Used for evaluating the model's performance on downstream tasks.
  • Penn Tree Bank dataset: Used for fine-tuning and analyzing token iterations on probing tasks.

The paper does not introduce new benchmarks but rather applies existing ones to analyze their proposed dynamic depth mechanism.

Key Insights and Ideas

  • Iterative Refinement: The core idea is that Transformer layers act as iterative steps that progressively refine token representations, rather than simply extracting distinct features at each fixed layer.
  • Token-Specific Processing: Not all tokens are created equal. The model learns to allocate more processing (iterations) to tokens that are more critical or difficult for the task at hand.
  • Special Tokens' Importance: The [CLS] token consistently receives more iterations, likely because it's used for tasks like SOP and needs to aggregate information for the entire input. Conversely, [SEP] tokens receive fewer iterations, aligning with observations of their "no-op" behavior in attention mechanisms.
  • MLM Task Complexity: Tokens masked with [MASK/MASK] or [MASK/random] receive more iterations than [MASK/original], suggesting these are harder for the model to identify.
  • Fixed Points: The iterative process, controlled by a convex combination of states, naturally leads to hidden states converging towards a "fixed point" for each token, indicating a stable representation is reached.
  • Task Complexity Adaptation: Downstream tasks that require more complex sentence structure inference (e.g., TopConst) lead to more token iterations compared to simpler tasks (e.g., Tense).

Key Results

  • Pre-training: The dynamic depth model achieves comparable MLM and SOP accuracy to standard ALBERT models, demonstrating that efficient iteration doesn't necessarily sacrifice performance.
  • Downstream Tasks (GLUE): ALBERT models with adaptive depth perform competitively on the GLUE benchmark, with the ALBERT-base adaptive depth model being very close to its fixed-depth counterpart. This suggests that dynamic depth can achieve similar performance with potentially fewer computations.
  • Probing Tasks: The model shows that tokens with specific dependency roles (e.g., subj for SubjNum, obj and root for ObjNum) receive more iterations, reinforcing the idea that crucial tokens are processed more.
  • Efficiency: While not explicitly quantified in terms of FLOPs reduction, the introduction of a ponder cost and the observed dynamic iteration suggests a mechanism for reducing computational load by avoiding unnecessary processing of simpler tokens.

Limitations and Future Work

  • Computational Cost: While the paper introduces a mechanism for efficiency, a direct quantitative comparison of computational cost (e.g., FLOPs) between fixed and adaptive depth models is not extensively detailed.
  • Hyperparameter Tuning: The ponder cost factor ($\gamma$) needs to be empirically calibrated, which adds a hyperparameter to tune.
  • Future Work: The authors hope their findings will contribute to a better understanding of Transformer convergence mechanisms, potentially leading to reduced computational footprints and new regularization methods. They suggest this iterative perspective could be a valuable lens for future research into Transformer architectures.

Summary updated

Abstract

Antoine Simoulin, Benoit Crabbé. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop. 2021.
BibTeX
@inproceedings{simoulin2021how,
  author = {Antoine Simoulin and Benoît Crabbé},
  title = {{How Many Layers and Why? An Analysis of the Model Depth in Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=NQOeNvi1Rj},
}
Figure