Abstract
With the increasing computational demands of neural networks, many hardware accelerators for the neural networks have been proposed. Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network primitive that enables neural networks to retrieve most relevant information from a knowledge-base, external memory, or past states. The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time. We observe today's practice of implementing this mechanism using matrix-vector multiplication is suboptimal as the attention mechanism is semantically a content-based search where a large portion of computations ends up not being used. Based on this observation, we design and architect A3, which accelerates attention mechanisms in neural networks with algorithmic approximation and hardware specialization. Our proposed accelerator achieves multiple orders of magnitude improvement in energy efficiency (performance/watt) as well as substantial speedup over the state-of-the-art conventional hardware.
Introduction: The "Attention Tax"
While convolutional (CNN) and recurrent (RNN) networks have seen a gold rush of hardware accelerators, the attention mechanism—the backbone of modern models like BERT and Transformers—has been largely ignored by hardware designers. This oversight is costly: in models like BERT, attention can account for over 35% of total inference time, and in memory-intensive models like KV-MemN2N, it can dominate the execution profile.
The fundamental issue is that attention is traditionally implemented as dense matrix-vector multiplication. This approach computes a similarity score for every possible target in a knowledge base. However, attention is semantically a content-based search. After the softmax operation, most of these scores become near-zero and contribute nothing to the final result. Computing them all is computationally wasteful.
Methods: Algorithmic Approximation and Hardware Specialization
The $A^3$ architecture introduces a hardware-software co-design that treats attention as a retrieval task rather than a dense linear algebra problem. It employs two main approximation stages to skip unnecessary work:
1. Candidate Selection (The Greedy Filter)
Instead of computing full dot products for every row in a key matrix, $A^3$ uses a greedy algorithm. It preprocesses the key matrix by sorting columns. When a query arrives, it identifies the most promising "candidate" rows using a priority queue, computing full scores only for these likely winners.
2. Post-Scoring Selection (The Significance Filter)
Even among candidates, some will be much stronger than others. $A^3$ identifies the top-scoring row and discards any candidates whose scores are significantly lower based on a configurable threshold. Since the subsequent softmax function is exponential, these lower scores would have vanished in the final output anyway.
Hardware Specialization
The $A^3$ architecture is a three-stage pipeline (Dot-Product, Exponent, and Output) designed to handle these approximations in parallel. It utilizes a custom 8-bit fixed-point quantization scheme to further reduce power consumption without sacrificing model accuracy.
Key Results
The researchers evaluated $A^3$ across three diverse natural language processing (NLP) workloads: Facebook bAbI QA (MemN2N), WikiMovies QA (KV-MemN2N), and SQuAD v1.1 (BERT).
- Energy Efficiency: $A^3$ achieves over $10^4\times$ better energy efficiency (performance/watt) than an Intel Xeon CPU and $10^3\times$ better than an NVIDIA Titan V GPU.
- Throughput and Latency: For BERT, 6–7 $A^3$ units provide higher throughput than a flagship Titan V GPU. The approximation schemes allow $A^3$ to process attention operations with significantly lower latency than standard hardware designs.
- Accuracy: Under a "conservative" configuration, $A^3$ maintains accuracy within 1% of the original model while still providing significant speedups.
Conclusion
By shifting the paradigm of attention from "multiply everything" to "search for the relevant," $A^3$ unlocks massive optimization potential. The design is "approximation-aware," allowing users to tune iteration counts and thresholds to trade off a negligible amount of accuracy for massive gains in speed and power. While preprocessing overhead remains a factor for dynamic keys, $A^3$ demonstrates that specialized hardware for attention is a critical path for the next generation of efficient AI.
Summary updated