Topkima-Former: Low-Energy, Low-Latency Inference for Transformers Using Top-k In-Memory ADC

Shuai Dong, Junyi Yang, Xiaoqi Peng, Hongyang Shang, Ye Ke, Xiaofeng Yang, Hongjie Liu, Arindam Basu

TL;DR

Topkima-Former accelerates Transformer inference by integrating top-$k$ selection directly into the In-Memory ADC (IMA) process. By using a novel "decreasing ramp" ADC that identifies the largest values first and stops early, the system eliminates sorting overhead and achieves up to 84× speedup and 35× energy efficiency gains over state-of-the-art accelerators with negligible accuracy loss.

What Problem Does This Paper Address?

While Transformers have revolutionized NLP and Computer Vision, they are notoriously "heavy" for hardware. Most research focuses on accelerating linear operations (Matrix Multiplications), but the non-linear softmax operation has become a massive bottleneck, accounting for up to 40% of total inference latency.

Existing In-Memory Computing (IMC) solutions—which process data directly within memory arrays to save energy—face two major hurdles:

  1. ADC Overhead: Converting analog signals from the memory array back to digital data (via Analog-to-Digital Converters) is slow and power-hungry.
  2. Sorting Complexity: Implementing "top-$k$" attention (processing only the most important values) usually requires a digital sorting step after conversion, which consumes significant time and area.

Methods

The authors propose a "circuit-architecture-algorithm" co-design called Topkima-Former:

  • Circuit Level (Topkima-M): The core innovation is a modified In-Memory ADC. Standard ramp ADCs count "up" from zero. Topkima uses a decreasing ramp. Because the ramp starts high, the largest Multiply-Accumulate (MAC) voltages trigger the sense amplifiers first. The system simply stops the ADC once $k$ values are found. This effectively performs "sorting" for free during the conversion process and allows for "early stopping," saving both time and power.
  • Algorithm Level (TFCBP): To minimize accuracy loss from only looking at $k$ values, the authors use Top-k Forward-Complete Backward Propagation. During training, only the top-$k$ values are used in the forward pass, but all values participate in the gradient calculation during the backward pass.
  • Architecture Level (Scale-Free Attention): Standard attention requires dividing by $\sqrt{d_k}$. Instead of performing this division in hardware, the authors "fold" this scaling factor into the weights ($W_Q$) offline, removing the hardware overhead entirely.

Datasets

The researchers evaluated Topkima-Former across several standard benchmarks:

  • Computer Vision: ViT (Vision Transformer) evaluated on CIFAR-10 and CIFAR-100.
  • NLP: DistilBERT and BERT-base evaluated on the SQuAD v1.1 (Stanford Question Answering Dataset).

Key Insights and Ideas

  • The "Aha!" Moment in ADC Design: By simply reversing the direction of the ADC ramp, the hardware naturally prioritizes the most "important" (largest) signals. This turns a serial bottleneck (ADC + Sorting) into a single, truncated operation.
  • SRAM vs. RRAM Hybridization: The design uses RRAM (Resistive RAM) for static projection weights due to its high density, but uses SRAM for the attention mechanism ($Q \cdot K^T$) because attention scores change with every input and require the high endurance of SRAM.
  • Sub-top-k Strategy: Because physical memory crossbars have size limits, the system performs "local" top-$k$ selection within sub-arrays. The authors found that even this fragmented selection maintains high accuracy while being much easier to implement in hardware.

Key Results

  • Massive Speedups: Topkima-Former achieves 1.8× to 84× speedup compared to prior IMC-based accelerators.
  • Energy Efficiency: It is 1.3× to 35× more energy-efficient than state-of-the-art competitors.
  • Softmax Performance: The specific "topkima" softmax macro is 15× faster than conventional softmax hardware and 8× faster than digital top-$k$ implementations.
  • High Accuracy: With $k=5$, the models showed only a 0.4% to 1.2% reduction in accuracy across all tested datasets, a very acceptable trade-off for the performance gains.
  • Throughput: The system achieves a throughput of 6.70 TOPS and an energy efficiency of 16.84 TOPS/W.

Limitations and Future Work

  • Crossbar Fragmentation: As mentioned in the paper, splitting the matrix across multiple crossbars means the hardware selects the "top-$k$ of each sub-array" rather than a true global top-$k$. While accuracy holds for current models, this might become a challenge for models requiring extremely sparse global attention.
  • Hardware Complexity: While it saves energy, the arbiter-encoder logic required to manage the "early stopping" across many columns adds some complexity to the peripheral circuitry.
  • Future Scaling: The authors suggest that these benefits will scale even better with larger sequence lengths (e.g., in models like GPT-3.5 with 4096 tokens), but they have yet to provide silicon-level measurements for those specific scales.

Summary updated

Abstract

Transformer model has gained prominence as a popular deep neural network architecture for neural language processing (NLP) and computer vision (CV) applications. However, the extensive use of nonlinear operations, like softmax, poses a performance bottleneck during transformer inference and comprises up to 40% of the total latency. Hence, we propose innovations at the circuit, architecture, and algorithm levels to accelerate the transformer. At the circuit level, we propose topkima-combining top-k activation selection with in-memory ADC (IMA) to implement a low-energy and low-latency softmax without any sorting latency. Only the k largest activations are sent to the softmax calculation block, reducing the huge computational cost of softmax. Using a modified training scheme with top-k only in the forward pass, experimental results demonstrate only a 0.4% to 1.2% reduction in accuracy across ViT, distilBERT, and BERT-base models when evaluated on CIFAR-10, CIFAR-100, and SQuAD datasets with k=5. At the architecture level, an improved scale-free technique is introduced to reduce the computational cost of attention. The combined system, dubbed Topkima-Former, enhances 1.8x-84x speedup and 1.3x-35x energy efficiency (EE) over prior In-memory computing (IMC) accelerators. Compared to a conventional softmax macro and a digital top-k (Dtopk) softmax macro, our proposed tokima softmax macro achieves about 15x and 8x faster speed respectively.
BibTeX
@article{dong2025topkima,
  author = {Shuai Dong and Junyi Yang and Xiaoqi Peng and Hongyang Shang and Ye Ke and Xiaofeng Yang and Hongjie Liu and Arindam Basu},
  title = {{Topkima-Former: Low-Energy, Low-Latency Inference for Transformers Using Top-k In-Memory ADC}},
  journal = {IEEE Trans. Circuits Syst. I Regul. Pap.},
  year = {2025},
  doi = {10.1109/tcsi.2025.3549060},
}

Loading related content...

Figure