Token Representation Collapse in Deep Transformers

115 concepts87 papers1 proposal

When we scale up Transformer architectures by stacking more layers, we expect the model to learn increasingly complex, hierarchical representations. However, deep Transformers often hit a performance ceiling where adding more layers yields diminishing returns or even degrades performance.

The root cause of this stagnation is a phenomenon known as token representation collapse (or over-smoothing). As data flows through a deep Transformer, individual token representations—whether they represent words in a sentence or patches in an image—lose their unique identity and converge toward a high degree of similarity. Over multiple layers, they become mathematically indistinguishable.

To understand why this happens and how to fix it, we must look at the geometric and mathematical operations that define the Transformer backbone.

The Intuition: Attention as an Iterative Averaging Filter

At its core, the self-attention mechanism updates a token's representation by taking a weighted average of all other tokens in the sequence. If you have a set of distinct vectors in space and you repeatedly replace each vector with a weighted average of its neighbors, the vectors will naturally contract toward each other.

Over many successive layers, this continuous averaging pushes token hidden states toward a central cluster. Instead of maintaining localized, fine-grained semantic or spatial features, every token begins to reflect a generalized, global context.

This behavior can be formally analyzed through the lens of signal processing. In Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice, Peihao Wang et al. provide a rigorous proof that self-attention acts fundamentally as a low-pass filter. It systematically dampens high-frequency variations (the unique, sharp details of individual tokens) and preserves only the low-frequency component (the global average, or "direct-current" component).

This smoothing effect is compounded by Layer Normalization (LayerNorm). In Revisiting Over-smoothing in BERT from the Perspective of Graph, Han Shi demonstrates that LayerNorm acts as a primary driver of rank collapse by projecting token representations into a low-rank subspace. The high scaling factors inherent to normalization accelerate this homogenization, stripping the network of its representational diversity.

The Consequences of Collapse

When token representations collapse, the model suffers from several distinct failure modes:

Standard Attention (Low-Pass Filter):
Input Tokens ---> [ Layer 1: Averaging ] ---> [ Layer N: Uniformity ] ---> Rank Collapse (Information Loss)

Anti-Oversmoothing Mechanisms:
Input Tokens ---> [ High-Frequency Boost / Residual Shortcuts / Negative Weights ] ---> Diverse, Expressive Features

Countermeasures: Preserving Token Individuality

To prevent representations from melting into a uniform blur, recent research introduces mechanisms that either break the non-negative averaging constraint of attention, inject control loops, or preserve high-frequency details.

1. Modifying the Attention Operator

If standard SoftMax attention forces a non-negative weighted average that smooths out features, one solution is to allow the attention mechanism to differentiate or subtract.

In More Expressive Attention with Negative Weights, Ang Lv et al. introduce "Cog Attention," which allows attention weights to be negative. This enables attention heads to perform operations like deletion and refinement rather than just aggregation, maintaining token distinctness.

Another approach is to explicitly recycle the information lost during smoothing. Laziz Abdullaev introduces Transformer Meets Twicing: Harnessing Unattended Residual Information, which adapts a classical statistical technique called "kernel twicing" to capture and re-add the residual information filtered out by attention's low-pass nature.

Alternatively, Tam Minh Nguyen et al. look to control theory in PIDformer: Transformer Meets Control Theory, integrating a Proportional-Integral-Derivative (PID) controller into the attention block to prevent token representations from converging to a single steady-state vector.

2. Frequency-Domain Interventions

Since over-smoothing acts as a low-pass filter, models can be designed to explicitly amplify high-frequency components.

Peihao Wang et al. propose two plug-and-play scaling factors, AttnScale and FeatScale, which dynamically regulate the frequency spectrum of the feature maps to preserve high-frequency spatial details.

Similarly, when reducing token counts for efficiency, Dong-Jae Lee et al. design a strategy in Frequency-Aware Token Reduction for Efficient Vision Transformer that isolates and protects high-frequency tokens (those containing unique, detailed information) while aggregating low-frequency background tokens into a single representative vector.

3. Architectural Shortcuts and Value Residuals

A structural way to preserve token identity is to ensure that early, un-smoothed token information is directly accessible to deeper layers.

In Value Residual Learning, the authors introduce a shortcut that passes value vectors from the very first layer directly into deeper attention blocks. This prevents token-specific local context from being entirely diluted into sequence-level features.

On the optimization side, Lorenzo Noci et al. prove that applying a depth-dependent scaling factor ($1/\sqrt{L}$) to the residual branches counteracts the low-rank projection of LayerNorm, preserving token variance and allowing deep networks to be trained successfully even with basic SGD.

Key Foundational Work

Augmented Shortcuts for Vision Transformers (2021) — contributed 2 concept elements to this direction.
Adaptive Bi-directional Attention: Exploring Multi-Granularity Representations for Machine Reading Comprehension (2020) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_representation_over_smoothing_f5204760,
  title = {representation over-smoothing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representation-over-smoothing/rdr_f5204760c3b74c6ebe4610742ce251f6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_representation_over_smoothing_f5204760_v1,
  title = {representation over-smoothing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/representation-over-smoothing/rdr_f5204760c3b74c6ebe4610742ce251f6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake