Preserving Token Diversity in Deep Transformers

This direction is merged into another direction: representation over-smoothing. Its last version stays readable and citable here.

Version 1 of 137 papers44 conceptsfrozen 2026-10-04

In the quest for more powerful artificial intelligence, the standard recipe has been simple: add more layers. However, deep Transformer architectures—the backbone of modern NLP and computer vision—hit a mysterious wall as they scale. Instead of learning increasingly complex hierarchies, the representations of different tokens (words in a sentence or patches in an image) begin to converge.

This phenomenon, known as over-smoothing or representation collapse, results in a state where every token embedding looks nearly identical. When "apple" and "orange" become indistinguishable in the final layers, the model loses its ability to perform fine-grained reasoning, effectively capping the useful depth of the network.

The Low-Pass Filter Problem

To understand why over-smoothing happens, we must look at the mathematical heart of the Transformer: the self-attention mechanism. At its core, self-attention computes a weighted average of token features.

As Hyowon Wi and others have demonstrated through Fourier domain analysis, this averaging process acts as a low-pass filter. It smooths out the "high-frequency" signals—the unique, sharp details that distinguish one token from another—and preserves only the "low-frequency" global context. While some smoothing is necessary for context-aware representations, repeated applications across dozens of layers eventually wash away all local information.

Han Shi and colleagues further identified that Layer Normalization can exacerbate this issue by forcing embeddings into a similar range, accelerating the convergence toward a uniform state.

Consequences Across Domains

The impact of over-smoothing varies by task, but the result is always a loss of discriminative power:

  • Computer Vision: In Vision Transformers (ViTs), Daquan Zhou notes that over-smoothing causes distinct image patches to merge into a "fruit-colored blur." This is particularly damaging for tasks like medical segmentation, where Yaowu Hu found that "token uniformity" makes it nearly impossible for models to distinguish between the subtle boundaries of different organs.
  • Tabular Data: Structured data relies on maintaining precise, high-frequency relationships between features. Jayoung Kim argues that over-smoothing is the primary reason Transformers have historically struggled to beat tree-based models like XGBoost on tables.
  • Natural Language Processing: For tasks like entity retrieval or reading comprehension, Nuo Chen highlights that "homogeneity" in later layers prevents models from distinguishing between specific names or numbers, leading to errors in span extraction.

Strategies for Maintaining Diversity

Researchers are developing several distinct approaches to "un-smooth" the Transformer and allow for deeper, more expressive architectures.

1. Refining the Attention Mechanism

Rather than allowing attention to be a simple averager, new modules inject "sharpness" back into the process. The Refiner module by Daquan Zhou uses local convolutions to enhance attention maps directly. Similarly, Laziz Abdullaev introduced Twicing Attention, which borrows a technique from nonparametric regression to "recycle" the high-frequency information that is typically discarded during the smoothing process.

2. Frequency-Aware Inductive Biases

If the problem is a low-pass filter, the solution is to re-introduce high-frequency signals. Hyowon Wi proposed Graph-Filter-based Self-Attention (GFSA) and CheAtt, which use Chebyshev polynomials to allow the model to learn both low- and high-frequency filters. This allows the model to capture global context and local detail simultaneously.

3. Architectural and Training Regularization

Sometimes the fix is in how we build or train the model.

  • Value Residuals: Zhanchao Zhou introduced ResFormer, which uses "value residual connections" to provide a shortcut for raw token information from the first layer to reach the deepest parts of the network.
  • Diversity Objectives: Tianlong Chen proposed the Principle of Diversity, using specialized regularizers to penalize redundancy in patch embeddings and attention maps during training.
  • Self-Contrastive Learning: To combat "inter-layer over-smoothing," Nuo Chen developed SSCL, which uses a model's own intermediate layers as negative examples, forcing the final output to remain distinct and informative.

Rethinking Model Shape

Finally, the fight against over-smoothing is changing our understanding of the "optimal" Transformer shape. Fuzhao Xue discovered that while standard training objectives (like those in BERT) are highly susceptible to over-smoothing, others like Masked Autoencoders (MAE) naturally resist it. This insight led to the development of Bamboo, a deeper and narrower architecture that achieves superior performance by leveraging the inherent diversity-preserving properties of certain training tasks.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_preserving_token_diversity_in_deep_trans_10e34c47,
  title = {Preserving Token Diversity in Deep Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preserving-token-diversity-in-deep-transformers/rdr_10e34c4748524914b7f32655d14ed621},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_preserving_token_diversity_in_deep_trans_10e34c47_v1,
  title = {Preserving Token Diversity in Deep Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preserving-token-diversity-in-deep-transformers/rdr_10e34c4748524914b7f32655d14ed621/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}