Empirically, the power of modern Transformers stems from depth. Large language models stack dozens or hundreds of layers to build abstract, hierarchical representations of language and vision. Yet, if you look under the hood of Transformer theory, a striking paradox emerges: almost every rigorous mathematical guarantee we have—whether it proves global convergence, explains in-context learning, or maps optimization landscapes—is strictly confined to single-layer or shallow architectures.
This "single-layer constraint" represents a massive gap between what we can prove and what we actually deploy. To understand why this constraint exists, we must look at how mathematical tractability breaks down the moment layers begin to interact.
The Analytical Sandbox of Single Layers
Why do theorists love single-layer Transformers? Because a single self-attention layer isolated from the rest of the network behaves like well-understood classical mathematical objects. Without the compounding effects of depth, researchers can treat self-attention as a direct optimization or statistical projection problem.
For instance, consider In-Context Learning (ICL), where a model learns to solve a task from a few examples in its prompt without weight updates. In a single-layer model, this process can be elegantly mapped to classical optimization algorithms. Shuai Li et al. proved in The Closeness of In-Context Learning and Weight Shifting for Softmax Regression that a single self-attention layer implements a data transformation mathematically equivalent to a step of gradient descent. Similarly, Liam Collins et al. showed in In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness that single-layer attention naturally adapts to the geometric smoothness (Lipschitzness) of the prompt's underlying function.
Single layers also allow us to view attention through the lens of classical statistics. Rachel S. Y. Teo et al. demonstrated in Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis that the query-key-value mechanism is fundamentally performing Kernel Principal Component Analysis (Kernel PCA). By leveraging convex duality, Arda Sahiner et al. in Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers recast the non-convex training of a single attention block into an interpretable, globally solvable convex optimization problem.
In this single-layer sandbox, we can even prove exactly how optimization occurs. Daniel Hsu showed in Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot that a one-layer Transformer trained via standard gradient descent efficiently learns to isolate and average specific tokens. Meanwhile, Yuchen Li et al. in How Transformers Learn Topic Structure: Towards a Mechanistic Understanding mapped out a clean "two-stage" learning process where value matrices capture topic clusters before attention refines them.
Why Stacking Layers Breaks the Mathematics
If single-layer models are so beautifully interpretable, why can't we just stack these proofs on top of each other? The reason is that depth introduces three distinct types of mathematical complexity that destroy the assumptions of single-layer theory:
1. Hierarchical Token Composition
In a single layer, tokens interact directly with one another based on static or simple contextual relationships. As Yuandong Tian et al. noted in Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer, single-layer models cannot capture how tokens compose hierarchically over time. In a deep network, Layer 1 might combine letters into words, Layer 2 combines words into phrases, and Layer 3 extracts semantic intent. This sequential composition means the input to Layer 2 is no longer the original data distribution, but a highly non-linear, moving transformation of it, making closed-form tracking impossible.
2. Signal Propagation and Inter-Layer Cancellation
In deep architectures, attention layers do not operate in isolation; they are interleaved with residual streams, layer normalization, and feed-forward (FF) blocks. Sho Yokoi demonstrated this friction in Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps, showing that while an individual FF block might amplify a specific linguistic pattern, the surrounding residual and normalization layers often act to suppress or cancel out these signals. Proving how a signal survives or mutates across dozens of these alternating amplification and cancellation cycles remains an open challenge.
3. Non-Convex Optimization Landscapes
While single-layer attention can sometimes be relaxed into convex problems, stacking layers introduces severe non-convexity. Volkan Cevher et al. in On the Convergence of Encoder-only Shallow Transformers and Bingqing Song et al. in Unraveling the Gradient Descent Dynamics of Transformers both highlight that standard convergence guarantees break down in deep networks due to the intricate mathematical complexity of layer-to-layer gradient interactions. The softmax mechanism, which is smooth in a single layer, can trap deep models in highly suboptimal local minima during backpropagation.
The Representational Chasm
This mathematical breakdown isn't just an inconvenience for theorists; it reflects a fundamental difference in what single-layer and multi-layer models can actually compute.
In Separations in the Representational Capabilities of Transformers and Recurrent Architectures, Satwik Bhattamishra et al. proved that adding just a second layer to a Transformer unlocks a massive leap in computational power. They showed that while single-layer Transformers find tasks like string matching or nearest neighbor search nearly impossible to scale, a two-layer Transformer solves them with ease.
This implies that the representational limits of single-layer models—such as the "triple-wise" bottlenecks identified by Clayton Hendrick Sanford et al. in Representational Strengths and Limitations of Transformers—may be completely bypassed by deep architectures.
Pushing Past the Two-Layer Frontier
To bridge this gap, theoretical machine learning is cautiously moving from one layer to two. The most prominent example of this is the study of induction heads—the circuits responsible for sophisticated pattern matching and in-context replication.
An induction head fundamentally cannot exist in a single layer; it requires a minimum of two layers to operate, where the first layer copies contextual information into a token's representation, and the second layer uses that context to look backward. In How Transformers Implement Induction Heads: Approximation and Optimization Analysis, Mingze Wang et al. provided a rigorous mathematical analysis of this two-layer circuit, explaining why models undergo a sudden, abrupt transition during training when these heads form.
Yet, even as we master two-layer dynamics, production-scale models operate with hundreds of layers. Until theoretical frameworks can account for the cumulative signal propagation, multi-head coordination, and non-linear interactions across deep stacks, our understanding of why deep Transformers work so well will remain an empirical wonder rather than a proven certainty.