Abstract
Transformer models have gained prominence as a popular deep neural network architecture for natural language processing (NLP) and computer vision (CV) applications. However, the extensive use of nonlinear operations, like softmax, poses a performance bottleneck during transformer inference and comprises up to 40% of the total latency. Hence, we propose innovations at the circuit, architecture, and algorithm levels to accelerate the transformer. At the circuit level, we propose topkima—combining top-k activation selection with in-memory ADC (IMA) to implement a low-energy and low-latency softmax without any sorting latency. Only the k largest activations are sent to the softmax calculation block, reducing the huge computational cost of softmax. Using a modified training scheme with top-k only in the forward pass, experimental results demonstrate only a 0.4% to 1.2% reduction in accuracy across ViT, distilBERT, and BERT-base models when evaluated on CIFAR-10, CIFAR-100, and SQuAD datasets with k=5. At the architecture level, an improved scale-free technique is introduced to reduce the computational cost of attention. The combined system, dubbed Topkima-Former, enhances 1.8x-84x speedup and 1.3x-35x energy efficiency (EE) over prior In-memory computing (IMC) accelerators. Compared to a conventional softmax macro and a digital top-k (Dtopk) softmax macro, our proposed tokima softmax macro achieves about 15x and 8x faster speed respectively.
Introduction
While Transformers have revolutionized NLP and Computer Vision, they are notoriously resource-intensive for hardware. Most research focuses on accelerating linear operations like Matrix Multiplications. However, the non-linear softmax operation has emerged as a massive bottleneck, accounting for up to 40% of total inference latency. Existing In-Memory Computing (IMC) solutions face hurdles in ADC overhead and sorting complexity for top-k attention, which usually requires a digital sorting step after conversion.
Methods
The authors propose Topkima-Former, a "circuit-architecture-algorithm" co-design to address these bottlenecks by integrating top-k selection directly into the In-Memory ADC process.
Circuit Level: Topkima-M
The core innovation is a modified In-Memory ADC. Standard ramp ADCs count "up" from zero. Topkima uses a decreasing ramp. Because the ramp starts high, the largest Multiply-Accumulate (MAC) voltages trigger the sense amplifiers first. The system stops the ADC once $k$ values are identified. This effectively performs "sorting" for free during the conversion process and allows for "early stopping," saving both time and power.
Algorithm Level: TFCBP
To minimize accuracy loss from only processing $k$ values, the authors use Top-k Forward-Complete Backward Propagation (TFCBP). During training, only the top-$k$ values are used in the forward pass, but all values participate in the gradient calculation during the backward pass to maintain model stability.
Architecture Level: Scale-Free Attention
Standard attention requires dividing by $\sqrt{d_k}$. Instead of performing this division in hardware, the authors "fold" this scaling factor into the weights ($W_Q$) offline, removing the hardware overhead entirely.
Results
The researchers evaluated Topkima-Former across several standard benchmarks: ViT on CIFAR-10/100, and DistilBERT/BERT-base on SQuAD v1.1.
Performance and Efficiency
Topkima-Former achieves 1.8× to 84× speedup and is 1.3× to 35× more energy-efficient than state-of-the-art IMC accelerators. The specific "topkima" softmax macro is 15× faster than conventional softmax hardware and 8× faster than digital top-$k$ implementations. The system achieves a throughput of 6.70 TOPS and an energy efficiency of 16.84 TOPS/W.
Accuracy Trade-offs
As shown in the evaluation results, with $k=5$, the models showed only a 0.4% to 1.2% reduction in accuracy across all tested datasets. This demonstrates that the top-k approximation is highly effective for Transformer inference, providing significant hardware gains with negligible performance impact.
Conclusion
Topkima-Former demonstrates that integrating top-k selection directly into the ADC process can eliminate the sorting bottleneck in Transformer inference. By combining this with scale-free attention and specialized training schemes, the system achieves significant improvements in throughput and energy efficiency, paving the way for more efficient Transformer deployments on edge hardware.
Summary updated