Abstract
The Transformer architecture has revolutionized computer vision, but it carries a heavy computational tax: the standard Softmax attention mechanism scales quadratically ($O(N^2)$) with the number of tokens. This makes high-resolution image processing and long-sequence modeling prohibitively expensive. In their paper, "Agent Attention: On the Integration of Softmax and Linear Attention," researchers from Tsinghua University and Kuaishou Technology propose a clever middle ground that offers the best of both worlds. By introducing a small set of "agent tokens" to act as intermediaries, the proposed mechanism reduces computational complexity to linear while maintaining the high expressive power of Softmax attention.
The Challenge of Quadratic Complexity
Standard Softmax attention is powerful because it captures global dependencies, but its $O(N^2)$ complexity means that doubling the image resolution quadruples the computation. To mitigate this, researchers previously turned to restricted receptive fields (like window attention in Swin Transformers) or linear attention. However, restricted fields can hurt long-range modeling, and traditional linear attention often suffers from low expressive power and poor feature diversity.
The authors identify a key redundancy: in an image, many query tokens (like pixels in a clear blue sky) require very similar information. Agent Attention exploits this redundancy to maintain a global receptive field without the quadratic cost.
Methodology: The Agent Token Paradigm
The core innovation is the Agent Token ($A$). Instead of every Query ($Q$) talking to every Key ($K$), the process is split into two Softmax-based steps:
- Agent Aggregation: A small number of agent tokens (often generated via pooling) act as queries to aggregate information from all Keys ($K$) and Values ($V$).
- Agent Broadcast: The original Queries ($Q$) then aggregate information from these enriched agent tokens.
Mathematically, the authors prove that this "quadruplet" paradigm $(Q, A, K, V)$ is equivalent to a generalized form of linear attention. To make this work in practice, the authors introduce the Agent Attention Module, which includes an Agent Bias (a positional encoding scheme) and a Diversity Restoration Module (DWC). The DWC is a depthwise convolution layer added to the output to compensate for the "feature smoothing" effect often seen in linear attention, ensuring the model maintains high feature diversity.
Visualizing the Workflow
By designing the number of agent tokens ($n$) to be much smaller than the number of image tokens ($N$), the complexity becomes $O(Nnd)$, which is effectively linear. The following figure illustrates how these agent tokens act as a bridge between the input queries and the global context.
Experimental Results and Key Insights
The authors evaluated Agent Attention across a broad range of standard benchmarks, including ImageNet-1K for classification, COCO for object detection, and ADE20K for semantic segmentation.
- Classification: Agent-PVT-L achieved 83.7% Top-1 accuracy on ImageNet, outperforming the standard PVT-L (81.7%) while being more efficient.
- Speed: On a CPU, Agent-based models ran 1.7x to 2.1x faster than their Softmax counterparts. On GPUs, they achieved 1.4x to 1.7x speedups.
- Detection: On the COCO dataset, Agent-PVT-T showed a massive improvement of +4.7 box AP over the baseline PVT-T.
- Zero-Shot Acceleration: Perhaps most impressively, Agent Attention can be swapped into pre-trained models like Stable Diffusion. Dubbed AgentSD, it speeds up generation by up to 1.84x and actually improves image quality (lower FID scores) without any fine-tuning, reducing common anatomical errors in generated images.
Conclusion and Future Work
Agent Attention provides a mathematical bridge between Softmax and linear attention, proving that efficiency does not have to come at the cost of global context. While performance depends on the number of agent tokens per stage—suggesting a "fewer in shallow, more in deep" design principle—the results are robust across tasks. Future directions include applying Agent Attention to video modeling and multi-modal foundation models, where extreme sequence lengths make linear complexity a necessity.
Summary updated