When we scale up Transformer architectures by stacking more layers, we expect the model to learn increasingly complex, hierarchical representations. However, deep Transformers often hit a performance ceiling where adding more layers yields diminishing returns or even degrades performance.
The root cause of this stagnation is a phenomenon known as token representation collapse (or over-smoothing). As data flows through a deep Transformer, individual token representations—whether they represent words in a sentence or patches in an image—lose their unique identity and converge toward a high degree of similarity. Over multiple layers, they become mathematically indistinguishable.
To understand why this happens and how to fix it, we must look at the geometric and mathematical operations that define the Transformer backbone.
The Intuition: Attention as an Iterative Averaging Filter
At its core, the self-attention mechanism updates a token's representation by taking a weighted average of all other tokens in the sequence. If you have a set of distinct vectors in space and you repeatedly replace each vector with a weighted average of its neighbors, the vectors will naturally contract toward each other.
Over many successive layers, this continuous averaging pushes token hidden states toward a central cluster. Instead of maintaining localized, fine-grained semantic or spatial features, every token begins to reflect a generalized, global context.
This behavior can be formally analyzed through the lens of signal processing. In Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice, Peihao Wang et al. provide a rigorous proof that self-attention acts fundamentally as a low-pass filter. It systematically dampens high-frequency variations (the unique, sharp details of individual tokens) and preserves only the low-frequency component (the global average, or "direct-current" component).
This smoothing effect is compounded by Layer Normalization (LayerNorm). In Revisiting Over-smoothing in BERT from the Perspective of Graph, Han Shi demonstrates that LayerNorm acts as a primary driver of rank collapse by projecting token representations into a low-rank subspace. The high scaling factors inherent to normalization accelerate this homogenization, stripping the network of its representational diversity.
The Consequences of Collapse
When token representations collapse, the model suffers from several distinct failure modes:
- Vanishing Gradients at Initialization: If tokens become identical too early, the network experiences "rank collapse." Lorenzo Noci et al. in Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse and Alessio Giorlandino et al. in Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation show that rank collapse causes the gradients for queries and keys to vanish at initialization, making deep models exceptionally difficult to train without complex adaptive optimizers.
- Lazy Deep Layers: In trained models, deep layers often exhibit near-perfect cosine similarity between successive states. As noted by Pengxiang Li et al. in Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN, this lack of representational change means the deeper layers are effectively "lazy"—they cease to perform meaningful transformations, wasting computational capacity.
- Loss of Explainability and Localization: For tasks that require identifying specific evidence, token uniformity is catastrophic. L. Yuan et al. show in Boosting Explainability through Selective Rationalization in Pre-trained Language Models that when tokens share a uniform global context, selection mechanisms can no longer isolate human-readable snippets of text as evidence. In computer vision, Mengqi Xue et al. demonstrate in ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition that global self-attention makes patch tokens statistically homogeneous, preventing interpretability methods from separating distinct object parts from background noise.
Standard Attention (Low-Pass Filter):
Input Tokens ---> [ Layer 1: Averaging ] ---> [ Layer N: Uniformity ] ---> Rank Collapse (Information Loss)
Anti-Oversmoothing Mechanisms:
Input Tokens ---> [ High-Frequency Boost / Residual Shortcuts / Negative Weights ] ---> Diverse, Expressive Features
Countermeasures: Preserving Token Individuality
To prevent representations from melting into a uniform blur, recent research introduces mechanisms that either break the non-negative averaging constraint of attention, inject control loops, or preserve high-frequency details.
1. Modifying the Attention Operator
If standard SoftMax attention forces a non-negative weighted average that smooths out features, one solution is to allow the attention mechanism to differentiate or subtract.
In More Expressive Attention with Negative Weights, Ang Lv et al. introduce "Cog Attention," which allows attention weights to be negative. This enables attention heads to perform operations like deletion and refinement rather than just aggregation, maintaining token distinctness.
Another approach is to explicitly recycle the information lost during smoothing. Laziz Abdullaev introduces Transformer Meets Twicing: Harnessing Unattended Residual Information, which adapts a classical statistical technique called "kernel twicing" to capture and re-add the residual information filtered out by attention's low-pass nature.
Alternatively, Tam Minh Nguyen et al. look to control theory in PIDformer: Transformer Meets Control Theory, integrating a Proportional-Integral-Derivative (PID) controller into the attention block to prevent token representations from converging to a single steady-state vector.
2. Frequency-Domain Interventions
Since over-smoothing acts as a low-pass filter, models can be designed to explicitly amplify high-frequency components.
Peihao Wang et al. propose two plug-and-play scaling factors, AttnScale and FeatScale, which dynamically regulate the frequency spectrum of the feature maps to preserve high-frequency spatial details.
Similarly, when reducing token counts for efficiency, Dong-Jae Lee et al. design a strategy in Frequency-Aware Token Reduction for Efficient Vision Transformer that isolates and protects high-frequency tokens (those containing unique, detailed information) while aggregating low-frequency background tokens into a single representative vector.
3. Architectural Shortcuts and Value Residuals
A structural way to preserve token identity is to ensure that early, un-smoothed token information is directly accessible to deeper layers.
In Value Residual Learning, the authors introduce a shortcut that passes value vectors from the very first layer directly into deeper attention blocks. This prevents token-specific local context from being entirely diluted into sequence-level features.
On the optimization side, Lorenzo Noci et al. prove that applying a depth-dependent scaling factor ($1/\sqrt{L}$) to the residual branches counteracts the low-rank projection of LayerNorm, preserving token variance and allowing deep networks to be trained successfully even with basic SGD.
Key Foundational Work
Augmented Shortcuts for Vision Transformers (2021) — contributed 2 concept elements to this direction.
Adaptive Bi-directional Attention: Exploring Multi-Granularity Representations for Machine Reading Comprehension (2020) — contributed 2 concept elements to this direction.