An Attention Free Transformer

Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Joshua M. SusskindOpenReview 2020 · OpenReview

Abstract

We introduce Attention Free Transformer (AFT), an efficient variant of Transformers that eliminates the need for dot product self attention. In an AFT layer, the key and value are first combined with a set of learned position biases, the result of which is multiplied with the query in an element-wise fashion. This new operation has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible to both large input and model sizes. We also introduce AFT-local and AFT-conv, two model variants that take advantage of the idea of locality and spatial weight sharing while maintaining global connectivity. We conduct extensive experiments on two autoregressive modeling tasks (CIFAR10 and Enwik8) as well as an image recognition task (ImageNet-1K classification). We show that AFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.

Introduction

Since their introduction, Transformers have revolutionized AI, but they come with a significant "tax": the self-attention mechanism. Standard self-attention requires calculating a compatibility matrix between every pair of elements in a sequence, resulting in quadratic complexity ($O(T^2)$) relative to sequence length ($T$). As sequences grow—such as in high-resolution images or long documents—the memory and processing requirements explode, limiting scalability.

While previous "linear attention" research attempted to solve this, many methods still suffered from quadratic complexity relative to the feature dimension ($d$) or required complex approximations that sacrificed expressiveness. The Attention Free Transformer (AFT) addresses this by replacing the resource-heavy dot-product mechanism with an efficient element-wise operation that maintains global connectivity while achieving linear memory complexity ($O(Td)$).

The AFT Mechanism

The core innovation of this architecture is the AFT layer. In a standard Transformer, one computes $Softmax(QK^T)V$. In AFT, the operations are rearranged to prioritize efficiency without losing the ability for any two points in a sequence to interact.

The AFT operation: element-wise interaction between query and a weighted context of keys and values with learned position biases.

As shown in the mechanism above, the process follows three steps:

  1. Weighted Context: For each target position, the keys ($K$) and values ($V$) are combined with a set of learned pair-wise position biases ($w$).
  2. Element-wise Interaction: This reduced "context" is multiplied by the query ($Q$) using a simple element-wise product rather than a matrix dot-product.
  3. Linear Complexity: By avoiding the $T \times T$ attention matrix, the memory footprint remains linear relative to both sequence length and feature dimension.

Model Variants

The authors introduced three main variants to tailor AFT to different tasks:

  • AFT-full: Provides global connectivity where every position can influence every other position.
  • AFT-local: Constrains position biases to a local window ($s$) to improve efficiency while maintaining a baseline level of global connectivity.
  • AFT-conv: Extends the local idea with spatial weight sharing (similar to CNNs), making it particularly effective for vision tasks and handling variable-sized inputs.

Experimental Results

AFT was evaluated across three distinct domains, demonstrating state-of-the-art efficiency and competitive accuracy:

  • Image Autoregressive Modeling (CIFAR-10): AFT-local achieved 2.74 bits/dim, outperforming standard Transformers while using only one-third of the memory and running significantly faster.
  • Language Modeling (Enwik8): AFT-local outperformed other efficient variants like Reformer and Performer in training loss, maintaining a much smaller memory footprint (approx. 11GB vs 20-30GB for competitors).
  • Image Classification (ImageNet-1K): AFT-conv reached 81.0% Top-1 accuracy, outperforming the popular DeiT-small (79.9%) with a similar parameter count.

Visualizing Learned Features

One of the most compelling insights from AFT is its ability to learn meaningful inductive biases. By visualizing the learned "keys" in AFT-conv, researchers observed that as depth increases, the model naturally evolves to recognize specific object parts or features, behaving much like the filters in a deep Convolutional Neural Network.

Visualizations of learned keys in AFT-conv, showing the evolution of feature detectors from simple patterns to complex object parts.

Furthermore, AFT can be pushed to extreme sparsity using Gumbel-Softmax. Even when each head is constrained to look at only a single relative position, the model maintains high accuracy (79.9% on ImageNet), suggesting that the architecture is inherently robust.

Conclusion and Future Work

AFT represents a significant step toward making Transformer-like architectures viable for massive sequences and resource-constrained environments. While it is not a direct mathematical approximation of dot-product attention—meaning pre-trained weights from models like GPT-3 cannot be directly loaded—it serves as a powerful "plugin replacement" for future training.

Future work may explore further optimizing AFT for mobile and edge devices through extreme sparsity and applying the AFT-conv logic to even larger-scale multimodal tasks.

Summary updated

Abstract

We introduce Attention Free Transformer (AFT), an efficient variant of Transformers \citep{transformer} that eliminates the need for dot product attention. AFT offers great simplicity and efficiency compared with standard Transformers, where the multi-head attention operation is replaced with the composition of element-wise multiplications/divisions and global/local pooling. During training time, AFT has linear time and space complexity w.r.t. both the sequence length and feature dimension; in the autoregressive decoding mode, AFT has constant memory and time complexity per step. We show that, surprisingly, we are able to train AFT effectively on challenging benchmarks, and also to match or surpass the standard Transformer counterparts and other efficient variants. In particular, AFT achieves the state-of-the-art result on CIFAR10 autoregressive modeling with much reduced complexity, and also outperforms several efficient Transformer variants on Enwik8.
BibTeX
@misc{zhai2020attention,
  author = {Shuangfei Zhai and Walter Talbott and Nitish Srivastava and Chen Huang and Hanlin Goh and Joshua M. Susskind},
  title = {{An Attention Free Transformer}},
  year = {2020},
  eprint = {2105.14103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.14103},
  howpublished = {OpenReview},
}
Figure