Abstract
The representation degeneration problem, specifically anisotropy, is a widely observed phenomenon in self-supervised Transformer models. Anisotropy refers to hidden representations being unexpectedly close in angular distance (high cosine similarity), effectively occupying a narrow cone in the vector space. While previous research attributed this to optimizing cross-entropy on long-tailed token distributions, this paper demonstrates that anisotropy persists in models with different objectives and across various modalities. Our findings suggest that anisotropy may be an inherent structural property of the Transformer architecture itself.
Introduction: The Representation Degeneration Problem
In Natural Language Processing (NLP), Transformers often fail to utilize the full dimensionality of their latent space. Instead, hidden states become anisotropic, pointing in a similar direction and leading to high average cosine similarity even between semantically unrelated inputs. The prevailing "softmax bottleneck" theory suggests this occurs because models must handle long-tailed distributions of rare tokens. This study investigates whether anisotropy is a fundamental characteristic of the Transformer architecture across modalities, rather than a mere artifact of language training.
Methodology
The authors employed a multi-pronged empirical strategy:
- Cross-Modality Analysis: Measuring average cosine similarity across layers for token-based NLP (BERT, GPT-2), character-based NLP (CANINE, ByT5), Speech (wav2vec 2.0, Whisper), and Vision (ViT, BEiT) models.
- Correlation Testing: Using Spearman and Pearson tests to determine if the "drift" (norm of the average representation) correlates with anisotropy levels.
- Untrained Block Experiments: Isolating architecture from data by analyzing untrained BERT blocks to observe how self-attention responds to manual input bias.
Key Insights: Beyond Rare Tokens
A critical finding is that anisotropy is not exclusive to subword-level models. Character-aware and byte-level models, which do not face the "rare token" problem in the same way, still exhibit significant anisotropy. For instance, the ByT5 decoder shows extremely high levels of representation clustering in its later layers.
The authors propose that anisotropy might be a necessary trade-off for "sharp" attention. To achieve categorical focus on specific input parts, the mathematical side effect is often a drift in representations that induces anisotropy.
Results Across Modalities
The study confirms that anisotropy is modality-agnostic, appearing in both Speech and Vision Transformers. In contrast, CNN-based vision models like ResNet remain isotropic, likely due to Batch Normalization centering the data.
- Speech: Models like HuBERT and Whisper maintain an average cosine similarity of at least 0.2 across almost every layer.
- Vision: While ViT and BEiT are anisotropic, the MiT model is a notable exception, remaining isotropic.
- Untrained Dynamics: Experiments with untrained blocks show that increasing input bias naturally leads to sharper softmax distributions, suggesting the architecture is "primed" for anisotropy even before training.
Conclusion and Future Work
The evidence suggests that the Transformer's self-attention mechanism inherently encourages anisotropic representations. While "drift" is a major factor in subword NLP models, its correlation is weaker in speech models, pointing to deeper structural causes. Future work should investigate how anisotropy evolves during pre-training and explore architectural modifications that allow for sharp attention without sacrificing latent space efficiency.
Summary updated