Is Anisotropy Inherent to Transformers?

Nathan Godey, Éric de la Clergerie, Benoît SagotarXiv 2023 · OpenReview

Abstract

The representation degeneration problem, specifically anisotropy, is a widely observed phenomenon in self-supervised Transformer models. Anisotropy refers to hidden representations being unexpectedly close in angular distance (high cosine similarity), effectively occupying a narrow cone in the vector space. While previous research attributed this to optimizing cross-entropy on long-tailed token distributions, this paper demonstrates that anisotropy persists in models with different objectives and across various modalities. Our findings suggest that anisotropy may be an inherent structural property of the Transformer architecture itself.

Introduction: The Representation Degeneration Problem

In Natural Language Processing (NLP), Transformers often fail to utilize the full dimensionality of their latent space. Instead, hidden states become anisotropic, pointing in a similar direction and leading to high average cosine similarity even between semantically unrelated inputs. The prevailing "softmax bottleneck" theory suggests this occurs because models must handle long-tailed distributions of rare tokens. This study investigates whether anisotropy is a fundamental characteristic of the Transformer architecture across modalities, rather than a mere artifact of language training.

Methodology

The authors employed a multi-pronged empirical strategy:

  1. Cross-Modality Analysis: Measuring average cosine similarity across layers for token-based NLP (BERT, GPT-2), character-based NLP (CANINE, ByT5), Speech (wav2vec 2.0, Whisper), and Vision (ViT, BEiT) models.
  2. Correlation Testing: Using Spearman and Pearson tests to determine if the "drift" (norm of the average representation) correlates with anisotropy levels.
  3. Untrained Block Experiments: Isolating architecture from data by analyzing untrained BERT blocks to observe how self-attention responds to manual input bias.

Key Insights: Beyond Rare Tokens

A critical finding is that anisotropy is not exclusive to subword-level models. Character-aware and byte-level models, which do not face the "rare token" problem in the same way, still exhibit significant anisotropy. For instance, the ByT5 decoder shows extremely high levels of representation clustering in its later layers.

Figure 2: Average cosine similarity across layers for character-based and byte-level models, showing persistent anisotropy.

The authors propose that anisotropy might be a necessary trade-off for "sharp" attention. To achieve categorical focus on specific input parts, the mathematical side effect is often a drift in representations that induces anisotropy.

Results Across Modalities

The study confirms that anisotropy is modality-agnostic, appearing in both Speech and Vision Transformers. In contrast, CNN-based vision models like ResNet remain isotropic, likely due to Batch Normalization centering the data.

Figure 5: Average cosine similarity in (a) Speech and (b) Vision Transformers, demonstrating that anisotropy is not limited to text.
  • Speech: Models like HuBERT and Whisper maintain an average cosine similarity of at least 0.2 across almost every layer.
  • Vision: While ViT and BEiT are anisotropic, the MiT model is a notable exception, remaining isotropic.
  • Untrained Dynamics: Experiments with untrained blocks show that increasing input bias naturally leads to sharper softmax distributions, suggesting the architecture is "primed" for anisotropy even before training.

Conclusion and Future Work

The evidence suggests that the Transformer's self-attention mechanism inherently encourages anisotropic representations. While "drift" is a major factor in subword NLP models, its correlation is weaker in speech models, pointing to deeper structural causes. Future work should investigate how anisotropy evolves during pre-training and explore architectural modifications that allow for sharp attention without sacrificing latent space efficiency.

Summary updated

Abstract

The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations tend to demonstrate that anisotropy might actually be inherent to Transformers-based models.
BibTeX
@misc{godey2023is,
  author = {Nathan Godey and Éric de la Clergerie and Benoît Sagot},
  title = {{Is Anisotropy Inherent to Transformers?}},
  year = {2023},
  eprint = {2306.07656},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.07656},
}
Figure