Topkima-Former: Low-energy, Low-Latency Inference for Transformers using top-k In-memory ADC

Shuai Dong, Junyi Yang, Xiaoqi Peng, Hongyang Shang, Ye Ke, Xiaofeng Yang, Hongjie Liu, Arindam Basu

Abstract

Transformer models have gained prominence as a popular deep neural network architecture for natural language processing (NLP) and computer vision (CV) applications. However, the extensive use of nonlinear operations, like softmax, poses a performance bottleneck during transformer inference and comprises up to 40% of the total latency. Hence, we propose innovations at the circuit, architecture, and algorithm levels to accelerate the transformer. At the circuit level, we propose topkima—combining top-k activation selection with in-memory ADC (IMA) to implement a low-energy and low-latency softmax without any sorting latency. Only the k largest activations are sent to the softmax calculation block, reducing the huge computational cost of softmax. Using a modified training scheme with top-k only in the forward pass, experimental results demonstrate only a 0.4% to 1.2% reduction in accuracy across ViT, distilBERT, and BERT-base models when evaluated on CIFAR-10, CIFAR-100, and SQuAD datasets with k=5. At the architecture level, an improved scale-free technique is introduced to reduce the computational cost of attention. The combined system, dubbed Topkima-Former, enhances 1.8x-84x speedup and 1.3x-35x energy efficiency (EE) over prior In-memory computing (IMC) accelerators. Compared to a conventional softmax macro and a digital top-k (Dtopk) softmax macro, our proposed tokima softmax macro achieves about 15x and 8x faster speed respectively.

Introduction

While Transformers have revolutionized NLP and Computer Vision, they are notoriously resource-intensive for hardware. Most research focuses on accelerating linear operations like Matrix Multiplications. However, the non-linear softmax operation has emerged as a massive bottleneck, accounting for up to 40% of total inference latency. Existing In-Memory Computing (IMC) solutions face hurdles in ADC overhead and sorting complexity for top-k attention, which usually requires a digital sorting step after conversion.

Methods

The authors propose Topkima-Former, a "circuit-architecture-algorithm" co-design to address these bottlenecks by integrating top-k selection directly into the In-Memory ADC process.

Topkima-Former system architecture overview showing the integration of RRAM and SRAM IMC.

Circuit Level: Topkima-M

The core innovation is a modified In-Memory ADC. Standard ramp ADCs count "up" from zero. Topkima uses a decreasing ramp. Because the ramp starts high, the largest Multiply-Accumulate (MAC) voltages trigger the sense amplifiers first. The system stops the ADC once $k$ values are identified. This effectively performs "sorting" for free during the conversion process and allows for "early stopping," saving both time and power.

Algorithm Level: TFCBP

To minimize accuracy loss from only processing $k$ values, the authors use Top-k Forward-Complete Backward Propagation (TFCBP). During training, only the top-$k$ values are used in the forward pass, but all values participate in the gradient calculation during the backward pass to maintain model stability.

Architecture Level: Scale-Free Attention

Standard attention requires dividing by $\sqrt{d_k}$. Instead of performing this division in hardware, the authors "fold" this scaling factor into the weights ($W_Q$) offline, removing the hardware overhead entirely.

Results

The researchers evaluated Topkima-Former across several standard benchmarks: ViT on CIFAR-10/100, and DistilBERT/BERT-base on SQuAD v1.1.

Accuracy comparison across different models and datasets for varying top-k values.

Performance and Efficiency

Topkima-Former achieves 1.8× to 84× speedup and is 1.3× to 35× more energy-efficient than state-of-the-art IMC accelerators. The specific "topkima" softmax macro is 15× faster than conventional softmax hardware and 8× faster than digital top-$k$ implementations. The system achieves a throughput of 6.70 TOPS and an energy efficiency of 16.84 TOPS/W.

Accuracy Trade-offs

As shown in the evaluation results, with $k=5$, the models showed only a 0.4% to 1.2% reduction in accuracy across all tested datasets. This demonstrates that the top-k approximation is highly effective for Transformer inference, providing significant hardware gains with negligible performance impact.

Conclusion

Topkima-Former demonstrates that integrating top-k selection directly into the ADC process can eliminate the sorting bottleneck in Transformer inference. By combining this with scale-free attention and specialized training schemes, the system achieves significant improvements in throughput and energy efficiency, paving the way for more efficient Transformer deployments on edge hardware.

Summary updated

Abstract

Transformer model has gained prominence as a popular deep neural network architecture for neural language processing (NLP) and computer vision (CV) applications. However, the extensive use of nonlinear operations, like softmax, poses a performance bottleneck during transformer inference and comprises up to 40% of the total latency. Hence, we propose innovations at the circuit, architecture, and algorithm levels to accelerate the transformer. At the circuit level, we propose topkima-combining top-k activation selection with in-memory ADC (IMA) to implement a low-energy and low-latency softmax without any sorting latency. Only the k largest activations are sent to the softmax calculation block, reducing the huge computational cost of softmax. Using a modified training scheme with top-k only in the forward pass, experimental results demonstrate only a 0.4% to 1.2% reduction in accuracy across ViT, distilBERT, and BERT-base models when evaluated on CIFAR-10, CIFAR-100, and SQuAD datasets with k=5. At the architecture level, an improved scale-free technique is introduced to reduce the computational cost of attention. The combined system, dubbed Topkima-Former, enhances 1.8x-84x speedup and 1.3x-35x energy efficiency (EE) over prior In-memory computing (IMC) accelerators. Compared to a conventional softmax macro and a digital top-k (Dtopk) softmax macro, our proposed tokima softmax macro achieves about 15x and 8x faster speed respectively.
BibTeX
@article{dong2025topkima,
  author = {Shuai Dong and Junyi Yang and Xiaoqi Peng and Hongyang Shang and Ye Ke and Xiaofeng Yang and Hongjie Liu and Arindam Basu},
  title = {{Topkima-Former: Low-energy, Low-Latency Inference for Transformers using top-k In-memory ADC}},
  journal = {IEEE Trans. Circuits Syst. I Regul. Pap.},
  year = {2025},
  doi = {10.1109/tcsi.2025.3549060},
}
Figure