Elliptical Attention

Stefan K. Nielsen, Laziz U. Abdullaev, Rachel S. Y. Teo, Tan M. Nguyen

TL;DR

This paper introduces "Elliptical Attention," a novel self-attention mechanism for transformers that uses a Mahalanobis distance instead of Euclidean distance to compute attention weights. This allows the model to focus more on contextually relevant information, leading to improved robustness against noisy data and reduced representation collapse, while achieving state-of-the-art performance across various tasks.

What Problem Does This Paper Address?

Transformers, particularly their self-attention mechanism, have revolutionized fields like NLP and computer vision. The standard self-attention relies on dot products between query and key vectors, effectively using Euclidean distance to determine attention weights. While powerful, this approach has two key weaknesses:

  1. Representation Collapse: The model can become overly reliant on a small subset of features, leading to a loss of diversity in learned representations. This can hinder generalization and make the model less capable of capturing nuanced relationships.
  2. Vulnerability to Contaminated Samples: Euclidean distance treats all dimensions of the feature space equally. This means that noisy or irrelevant features can disproportionately influence attention weights, making the model susceptible to errors when encountering slightly altered or corrupted input data.

The authors aim to address these limitations by developing a more sophisticated way to measure the relevance between tokens, one that can adapt to the underlying structure of the data.

Methods

The core innovation is Elliptical Attention, which replaces the standard Euclidean distance with a Mahalanobis distance for computing attention weights. Here's how it works:

  • Mahalanobis Distance: Instead of treating all feature dimensions equally, the Mahalanobis distance uses a learned metric (represented by a diagonal matrix M) to stretch or shrink the feature space along different dimensions. This allows the model to prioritize dimensions that are more informative for contextual relevance.
  • Hyper-Ellipsoidal Neighborhoods: By using the Mahalanobis distance, Elliptical Attention effectively defines hyper-ellipsoidal neighborhoods around queries. This means that tokens lying in "important" directions (where the underlying function varies more) are given more weight, while tokens in less important directions are down-weighted.
  • Estimating Coordinate-wise Variability: The key to constructing the Mahalanobis distance is estimating the "coordinate-wise variability" of the underlying function. The paper proposes an efficient, parameter-free estimator for this variability by looking at the difference in value vectors across consecutive layers. This avoids the need for learnable parameters or complex computations.
  • Theoretical Foundation: The authors connect their approach to non-parametric kernel regression, showing that using hyper-ellipsoidal neighborhoods can provably reduce variance in estimators without introducing bias, leading to improved robustness and less representation collapse.

Datasets

The paper evaluates Elliptical Attention on a variety of standard benchmarks:

  • Language Modeling: WikiText-103
  • Image Classification: ImageNet-1K
  • Long Sequence Modeling: LRA benchmark (including ListOps, Text, Retrieval, Image, Pathfinder)
  • Image Segmentation: ADE20K

Key Insights and Ideas

  • Beyond Euclidean Distance: The paper demonstrates that simply using Euclidean distance in self-attention is a significant limitation. By adapting the distance metric to the data's structure, we can unlock better performance.
  • Mahalanobis Distance as a Key: The Mahalanobis distance, by incorporating a learned metric, provides a more nuanced way to measure similarity, allowing attention to focus on contextually relevant features.
  • Stretching Feature Space: The idea of "stretching" the feature space along important directions is a powerful intuition for how Elliptical Attention achieves its benefits.
  • Unifying Robustness and Representation Collapse: The paper offers a theoretical framework that links representation collapse and robustness to the geometry of the attention mechanism's predictive neighborhood.
  • Efficiency is Key: The proposed estimator for coordinate-wise variability is efficient and requires no learnable parameters, making Elliptical Attention practical to implement without significant computational overhead.

Key Results

  • Robustness: Elliptical Attention achieves top or second-best perplexity on WikiText-103 under Word Swap contamination, outperforming baselines. On ImageNet, it shows competitive clean accuracy and significantly improved robustness against adversarial attacks (PGD, FGSM, SPSA, Auto Attack), often outperforming specialized robust models.
  • Representation Collapse: Experiments on ImageNet and ADE20K show that Elliptical Attention resists representation collapse, leading to more diverse token representations (Figure 6).
  • Efficiency: Elliptical Attention offers comparable or even better efficiency (speed and memory) than baseline robust models, making it a practical choice. For example, it shows no perceptible slowdown compared to DeiT.
  • Task Performance: The method demonstrates strong performance across various tasks, including object classification, image segmentation, and language modeling.

Limitations and Future Work

  • Noise Sensitivity of Estimator: The authors acknowledge that their estimator for coordinate-wise variability can be prone to noise, especially when using values from layers. They are exploring more precise estimation methods that maintain efficiency.
  • OOD Robustness: The paper notes that Elliptical Attention is designed for data that is shifted relative to the training distribution, not fundamentally different (Out-of-Distribution). While it shows some improvement on OOD benchmarks, it's not its primary strength.
  • Future Work: The authors suggest exploring more precise estimation methods for coordinate-wise variability and further investigating the theoretical underpinnings of their approach.

Summary updated

Abstract

Pairwise dot-product self-attention is key to the success of transformers that achieve state-of-the-art performance across a variety of applications in language and vision. This dot-product self-attention computes attention weights among the input tokens using Euclidean distance, which makes the model prone to representation collapse and vulnerable to contaminated samples. In this paper, we propose using a Mahalanobis distance metric for computing the attention weights to stretch the underlying feature space in directions of high contextual relevance. In particular, we define a hyper-ellipsoidal neighborhood around each query to increase the attention weights of the tokens lying in the contextually important directions. We term this novel class of attention Elliptical Attention. Our Elliptical Attention provides two benefits: 1) reducing representation collapse and 2) enhancing the model's robustness as the Elliptical Attention pays more attention to contextually relevant information rather than focusing on some small subset of informative features. We empirically demonstrate the advantages of Elliptical Attention over the baseline dot-product attention and state-of-the-art attention methods on various practical tasks, including object classification, image segmentation, and language modeling across different data modalities.
BibTeX
@inproceedings{nielsen2024elliptical,
  author = {Stefan K. Nielsen and Laziz U. Abdullaev and Rachel S. Y. Teo and Tan M. Nguyen},
  title = {{Elliptical Attention}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
}
Figure