Anisotropy is Not Inherent to Transformers

Anemily Machina, Robert E. MercerNAACL 2024 · DOI

TL;DR

This paper challenges the long-held assumption that Transformer language models are inherently anisotropic, meaning their internal representations are unevenly distributed. By analyzing the large Pythia model suite, the authors demonstrate that these models exhibit surprisingly isotropic embeddings, suggesting anisotropy might not be an unavoidable consequence of standard language modeling tasks. This finding opens new avenues for understanding and potentially mitigating representation degradation in Transformers.

What Problem Does This Paper Address?

For years, researchers have observed that Transformer models, despite their impressive performance, suffer from a phenomenon called "representation degradation" or anisotropy. This means the internal representations (embeddings) learned by the model are not uniformly distributed in space. Instead, they tend to cluster along certain directions, leading to an "anisotropic" embedding space. This anisotropy has been linked to degraded performance on downstream tasks and has been widely assumed to be an inherent limitation of Transformer architectures and standard language modeling objectives. The prevailing belief was that this was an unavoidable consequence, and much research focused on mitigating its effects rather than questioning its fundamental nature. This paper directly addresses this assumption by seeking counterexamples and investigating the conditions under which isotropy might be achieved.

Methods

The core of this research involves a multi-faceted analysis of isotropy in Transformer models, primarily focusing on the Pythia suite. The authors employ several established metrics to quantify isotropy:

  • Cosine Similarity (Inter-Sim and Intra-Sim): This measures the uniformity of directions between representation vectors. Inter-Sim looks at the average similarity between all pairs of vectors in a layer, while Intra-Sim focuses on the similarity for a specific token across different contexts.
  • Partition Function (I(W)): This metric, adapted from previous work, assesses the uniformity of distances of representations from the origin. An isotropic space would yield a constant partition function.
  • Layer Normalization Analysis: The authors specifically investigate the parameters of the final Layer Normalization (LayerNorm) layer, as it's a critical component in Transformer architectures. They analyze the norms of the learnable parameters (g and b) within LayerNorm, hypothesizing that these might play a role in achieving isotropy.
  • 2D SVD Projections: To visualize the embedding space, Singular Value Decomposition (SVD) is used to project high-dimensional embeddings into a 2D space, allowing for qualitative inspection of their distribution.

Crucially, the study analyzes models at various stages of training (using 21 evenly spaced checkpoints) and compares them to other prominent Transformer architectures.

Datasets

The primary dataset used for training and analysis is The Pile, a large, diverse 825GB English language dataset. The authors select specific text sources from The Pile that have a raw size of less than 10GB to manage computation time. They also utilize the SentEval Toolkit and the Language Model Evaluation Harness for evaluating downstream task performance on various classification tasks.

Key Insights and Ideas

  • Large Pythia Models are Isotropic: The most striking finding is that the larger Pythia models (410M parameters and above) exhibit significantly higher isotropy than previously observed in Transformer models, challenging the notion that anisotropy is inherent.
  • Untied Embeddings May Boost Isotropy: The Pythia models feature untied embedding and unembedding matrices, a design choice that the authors hypothesize contributes to their increased isotropy, especially in larger models.
  • LayerNorm Optimizes for Isotropy: Contrary to expectations, the study suggests that large Pythia models actively optimize their final LayerNorm layer for isotropy, rather than this layer contributing to anisotropy.
  • Anisotropy Doesn't Always Degrade Performance Steadily: The paper finds that anisotropy doesn't necessarily develop monotonically during training. Some models show an initial increase in isotropy before potentially becoming anisotropic, and the correlation between isotropy and downstream task performance is not always straightforward, especially for larger models.
  • Visualizations Reveal Unexpected Structures: 2D SVD projections of embedding spaces reveal that while randomly initialized models are isotropic, some trained models (like Pythia-70M) show projections consistent with previous theories of anisotropy (narrow cones), while larger Pythia models exhibit different structures, suggesting that non-visualized dimensions play a significant role.

Key Results

  • The Pythia models with at least 410M parameters show the highest isotropy scores across multiple metrics, with I(W) values reaching up to 0.82, significantly higher than previous benchmarks (around 0.52).
  • Mean centering Pythia models' embeddings consistently improves isotropy, indicating that while a translation might exist, the core representations are inherently more uniform.
  • For the Pythia-70M model, a sharp drop in isotropy is observed during the latter half of training, correlating with a decrease in downstream task performance.
  • The final LayerNorm parameters (g and b) in isotropic Pythia models tend to have smaller norms, particularly the 'b' parameter, and a smaller ratio of ||b||²/||g||² compared to anisotropic models.
  • The study found no significant correlation between the existence of "outlier dimensions" (identified by some previous work) and isotropy.

Limitations and Future Work

  • Ablation Study Needed: While the authors strongly suspect untied embedding matrices are key to Pythia's isotropy, they acknowledge that a formal ablation study is needed to confirm this causal link.
  • Theoretical Justification for LayerNorm Optimization: The paper provides empirical evidence that large Pythia models optimize LayerNorm for isotropy but lacks a theoretical explanation for why this occurs in these models and not others.
  • Masked Language Models (MLMs): The authors note that they haven't found globally isotropic examples for models trained with Masked Language Modeling (like BERT), suggesting this might be a distinct challenge.
  • Further Analysis: Future work could explore bias and clustering in these isotropic models and conduct a larger-scale ablation study to confirm the role of untied embeddings. The paper also suggests investigating the phase change from isotropy to anisotropy during training more deeply.

Summary updated

BibTeX
@inproceedings{machina2024anisotropy,
  author = {Anemily Machina and Robert E. Mercer},
  title = {{Anisotropy is Not Inherent to Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.274},
}
Figure