Attention: Self-Expression Is All You Need

Rene VidalICLR 2022 · OpenReview

TL;DR

This paper argues that the success of modern attention mechanisms, particularly in Transformers, isn't entirely novel. Instead, it traces attention's core ideas back to established techniques in manifold learning and image processing, like kernel regression and non-local means. The authors propose that the true innovations lie in the engineering of attention: its many learnable parameters, multiple heads, and deep stacking.

What Problem Does This Paper Address?

The paper tackles the question of what makes attention mechanisms, especially self-attention in Transformers, so powerful. While these models have revolutionized NLP and CV, their underlying principles are often presented as entirely new. This work aims to demystify attention by connecting it to a rich history of prior research, showing that many of its fundamental concepts have been explored before in different contexts. By understanding these connections, we can better appreciate the specific contributions of modern attention architectures and identify avenues for future improvement.

Methods

The authors systematically analyze attention mechanisms by drawing parallels with several established techniques:

  • Kernel Regression: They demonstrate that scaled dot-product attention is mathematically equivalent to kernel regression with a Gaussian kernel. The query, keys, and values in attention correspond to the input, kernel inputs, and output values in regression, respectively.
  • Non-Local Means Denoising: This image processing technique is shown to be a form of self-attention. The algorithm finds similar image patches (keys and queries) to denoise a central pixel (value), mirroring the self-attention process.
  • Locally Linear Embedding (LLE): LLE's objective of reconstructing data points from their neighbors is reinterpreted as a masked local self-attention mechanism. The coefficients in LLE are seen as attention weights, and the nearest neighbor constraint acts as a mask.
  • Subspace Clustering and Self-Expressiveness: The paper highlights the strong connection between self-attention and the concept of self-expressiveness, where data points are represented as linear combinations of other points. They argue that self-expressiveness provides a principled way to handle large numbers of tokens and that sparse regularization in this context is a precursor to sparse attention.

The paper's method is primarily analytical and comparative, dissecting the mathematical formulations of attention and its historical counterparts to reveal underlying similarities and differences.

Datasets

Not discussed in the paper. The paper focuses on theoretical connections and principles rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Attention is Not Entirely New: Many core ideas of attention, such as capturing long-range interactions and adapting to input data, were present in earlier manifold learning and image processing techniques.
  • Kernel Regression as Attention: Scaled dot-product attention can be viewed as a form of kernel regression, where the softmax function and learned linear transformations for queries and keys are key differentiators.
  • Non-Local Means as Self-Attention: The non-local means algorithm for image denoising is a direct precursor to self-attention, demonstrating the concept of finding similar patches (keys/queries) to influence a central value.
  • LLE as Masked Local Attention: Locally Linear Embedding can be understood as a masked self-attention mechanism, where attention is restricted to a local neighborhood defined by nearest neighbors.
  • Self-Expressiveness and Sparse Attention: The concept of self-expressiveness in subspace clustering offers a principled approach to selecting relevant data points (tokens) for representation, akin to sparse attention.
  • The Real Innovations: The paper argues that the primary advancements of modern attention mechanisms lie not in the fundamental idea of attention itself, but in the engineering: the use of numerous learnable parameters, multiple attention heads, and the stacking of layers to create deep architectures.

Key Results

The paper's "results" are primarily conceptual and analytical. It establishes mathematical equivalences and conceptual links between attention mechanisms and prior art. The key takeaway is that the engineering of attention (learnable parameters, multi-head, deep stacking) is what has led to its remarkable success, rather than a completely novel underlying principle.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while LLE coefficients are learned directly, this makes it harder to compute them for new data compared to attention mechanisms with learned weight matrices.
  • It notes that existing attention mechanisms (like additive attention) might not always be expressible in terms of kernels.
  • The analogy between sparse coding updates and transformer layers is not perfect, particularly regarding the residual connection.
  • Future Work:
    • The authors suggest exploring the connections between sparse subspace clustering and transformers by "unrolling" sparse coding iterates, potentially leading to deep multi-layer subspace clustering models.
  • They propose using attention mechanisms to parametrize self-expressive coefficients, as suggested by recent work.
  • They also suggest that sparse regularization, as used in subspace clustering, could be a more principled way to handle large numbers of tokens than arbitrary local neighborhood restrictions in attention.

Summary updated

Abstract

Transformer models have achieved significant improvements in performance for various learning tasks in natural language processing and computer vision. Much of their success is attributed to the use of attention layers that capture long-range interactions among data tokens (such as words and image patches) via attention coefficients that are global and adapted to the input data at test time. In this paper we study the principles behind attention and its connections with prior art. Specifically, we show that attention builds upon a long history of prior work on manifold learning and image processing, including methods such as kernel-based regression, non-local means, locally linear embedding, subspace clustering and sparse coding. Notably, we show that self-attention is closely related to the notion of self-expressiveness in subspace clustering, wherein data points to be clustered are expressed as linear combinations of other points with global coefficients that are adapted to the data and capture long-range interactions among data points. We also show that heuristics in sparse self-attention can be studied in a more principled manner using prior literature on sparse coding and sparse subspace clustering. We thus conclude that the key innovations of attention mechanisms relative to prior art are the use of many learnable parameters, and multiple heads and layers.
BibTeX
@inproceedings{vidal2022attention,
  author = {Rene Vidal},
  title = {{Attention: Self-Expression Is All You Need}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=MmujBClawFo},
}
Figure