Limits to Depth Efficiencies of Self-Attention

Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, Amnon Shashua

Abstract

While deep learning folklore suggests that "deeper is always better," this research demonstrates that self-attention architectures (Transformers) eventually hit a "depth-inefficiency" wall. By combining theoretical analysis of "separation rank" with extensive empirical tests, the authors show that once a model reaches a certain depth relative to its width, adding more layers is no more effective than simply making the model wider—and can even be counterproductive. This work provides a quantitative framework for balancing depth and width in large-scale model design.

Introduction

In the development of Convolutional Neural Networks (ConvNets), the "arms race" was defined by depth; architectures evolved from the 8 layers of AlexNet to over 100 in ResNets because depth provided an exponential boost in expressiveness. However, the Transformer revolution has not followed this same trajectory. Despite models growing from millions to billions of parameters, their depth has increased only marginally. For instance, GPT-3 (175B parameters) is only four times deeper than the original BERT-Large, despite being nearly 600 times larger in total parameters.

This paper addresses a fundamental question: Is there an optimal way to balance depth and width in self-attention networks? While previous empirical work, such as the OpenAI scaling laws, suggested that total parameter count is the primary driver of performance, this research argues that the relationship between depth and width is significantly more nuanced.

Methodology

Theoretical Framework: Separation Rank

The authors employ "separation rank" to quantify a network's capacity to model complex dependencies between different parts of an input sequence. By analyzing a "linearized" version of self-attention (removing Softmax and ReLU), they isolated the core connectivity of the architecture. This analysis revealed two distinct regimes:

  1. Depth-Efficient Regime: In shallow models where $L < \log_3(\text{width})$, adding layers increases expressiveness double-exponentially compared to increasing width.
  2. Depth-Inefficient Regime: Once a model passes a specific depth threshold, the advantage of depth vanishes. In this regime, depth and width contribute equally to the model's capacity.

Empirical Validation

To validate these theoretical findings, the authors trained a series of decoder-only Transformers with depths ranging from 6 to 48 layers across various widths. These models were trained on a massive corpus including English Wikipedia, BookCorpus, and OpenWebText. To ensure results were driven by internal representation rather than vocabulary memorization, a small vocabulary size of 2,000 tokens was utilized.

Results and Key Insights

The Transition to Depth Inefficiency

The experiments clearly demonstrated the transition predicted by the theory. For small parameter budgets, deeper models outperformed shallower ones. However, as the parameter budget increased, shallower models eventually caught up. If a deep model is not wide enough to utilize its layers, its performance saturates.

Empirical validation of depth regimes. (a) Data from Kaplan et al. (2020) showing scaling. (b) This work identifying the transition from depth-efficient to depth-inefficient regimes as parameter count increases.

Self-attention is remarkably effective at mixing information across a sequence—so effective that it quickly saturates the available representation space (the width). Once information is fully mixed, additional layers provide no benefit unless the width is also increased to accommodate more complex information. The transition from efficiency to inefficiency occurs at a threshold that depends logarithmically on the width ($L \sim \log(\text{width})$).

Re-evaluating GPT-3 and Future Scaling

One of the most provocative claims of this work is that the 175B parameter GPT-3, with its 96 layers, is past the point of optimal depth. The authors' fit suggests that for its parameter budget, a model with approximately 80 layers and a larger width would have achieved better performance.

Table comparing standard GPT-3 model dimensions with the theoretically optimal configurations derived from this study's scaling analysis.

Looking forward, the authors project that a 1-Trillion parameter model should not simply be a "taller" version of GPT-3. Instead, it should maintain a depth of around 95 layers while expanding to a massive width of 30,000.

Discussion and Limitations

While the empirical results on "real" Transformers match the theory remarkably well, the theoretical proofs rely on the linearization of attention (removing non-linearities). A full theoretical treatment including ReLU and Softmax remains an open challenge. Furthermore, this study focuses on the capacity of the network (expressiveness) rather than trainability (optimization). It is possible that deeper models are harder to optimize, which might further shift the optimal point toward wider, shallower architectures.

Finally, the authors noted a "width-efficiency" phenomenon in very small parameter regimes where deeper models actually performed worse than shallow ones—a behavior not yet fully explained by the current theory. Despite these limitations, the paper provides a specific formula to calculate the "transition size" for any given depth, allowing architects to determine precisely when to stop deepening and start widening their models.

Summary updated

BibTeX
@inproceedings{levine2020limits,
  author = {Yoav Levine and Noam Wies and Or Sharir and Hofit Bata and Amnon Shashua},
  title = {{Limits to Depth Efficiencies of Self-Attention}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Ymaym1qR6tR},
}
Figure