VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers

Run Wang, Gamze Islamoglu, Andrea Belano, Viviane Potocnik, Francesco Conti, Angelo Garofalo, Luca Benini

Abstract

As hardware accelerators significantly reduce the latency of General Matrix-Matrix Multiplications (GEMMs), non-linear operations like Softmax have emerged as the primary performance bottleneck in Transformer-based models. This paper presents VEXP, a lightweight RISC-V ISA extension designed to accelerate Bfloat16 exponentiation. By utilizing a hardware-optimized approximation based on Schraudolph’s method combined with a piecewise polynomial correction, VEXP achieves a 162.7× speedup in Softmax latency with a negligible 1% increase in silicon area. Our evaluation across GPT and Vision Transformer (ViT) models demonstrates that VEXP provides significant end-to-end speedups and energy savings without requiring model re-training or quantization-aware fine-tuning.

1. Introduction: The Softmax Bottleneck

In the current landscape of Large Language Models (LLMs), General Matrix-Matrix Multiplication (GEMM) has traditionally been the primary focus for hardware optimization. However, as dedicated accelerators make GEMMs increasingly efficient, Amdahl’s Law dictates that the remaining non-linear components of the Transformer architecture—specifically Softmax—begin to dominate the total execution time.

In an optimized RISC-V system running GPT-3, Softmax can account for up to 70% of the total runtime following GEMM acceleration. The difficulty in accelerating Softmax stems from two factors: its non-pointwise nature (requiring global maximums and sums) and its reliance on transcendental exponentiation. Traditional hardware approaches often rely on large lookup tables or low-precision math that necessitates expensive model re-training. VEXP addresses these challenges through a hardware/software co-design that prioritizes both silicon efficiency and numerical accuracy.

2. The VEXP Architecture

The VEXP extension is integrated into the Snitch RISC-V compute cluster, employing a multi-stage approach to approximate the exponential function $e^x$ efficiently.

2.1 Hardware-Optimized Exponentiation

Rather than calculating $e^x$ via a computationally expensive Taylor series, VEXP utilizes a hardware-optimized version of Schraudolph’s method. This technique exploits the IEEE 754 floating-point format by manipulating exponent and mantissa bits directly through integer addition and shifts. To mitigate the accuracy issues inherent in this approximation, the researchers added a second stage: a piecewise polynomial correction for the mantissa. This hybrid approach ensures the result is accurate enough for high-performance LLM inference while maintaining a minimal hardware footprint.

2.2 ISA Extensions and Software Integration

The extension introduces two custom instructions to the RISC-V ISA:

  • FEXP: Scalar Bfloat16 exponentiation.
  • VFEXP: A SIMD (Single Instruction, Multiple Data) variant that processes four Bfloat16 values simultaneously within a 64-bit register.

To maximize throughput, the hardware is integrated into a cluster utilizing Stream Semantic Registers (SSR) and Floating-Point Repetition (FREP). These features allow the system to stream data to the VEXP units at peak capacity, effectively eliminating the CPU overhead associated with loop management and manual memory addressing.

3. Evaluation and Methodology

The authors evaluated VEXP using a variety of industry-standard benchmarks and models to ensure broad applicability:

  • Language Models: GPT-2 Small and GPT-3 XL (evaluated on WikiText-2 for perplexity and ArcEasy for accuracy).
  • Vision Models: ViT-Base and ViT-Huge (evaluated on ImageNet-1K and CIFAR-10).
  • Kernels: FlashAttention-2, the current state-of-the-art implementation for attention mechanisms.

4. Key Results

4.1 Performance and Efficiency

Compared to a baseline RISC-V cluster, the VEXP extension delivered transformative gains in both speed and power consumption:

  • Softmax Latency: Achieved a 162.7× reduction in latency.
  • Energy Consumption: Reduced Softmax energy usage by 74.3×.
  • FlashAttention-2: Integration into this kernel resulted in an 8.2× throughput improvement and 4.1× higher energy efficiency.

4.2 End-to-End Impact

On a 16-cluster system (128 cores), VEXP translated these kernel-level gains into significant full-model improvements:

  • GPT-2: 5.8× total speedup.
  • GPT-3 XL: 2.9× total speedup.
  • Total Energy: 3.6× reduction in total energy consumption across inference tasks.

4.3 Area and Accuracy

One of the most significant contributions of VEXP is its efficiency. The hardware logic required for the extension represents only a 1% increase in silicon area at the cluster level. Furthermore, the approximation is highly precise, with a mean relative error of only 0.14%. This allows pre-trained BF16 models to be deployed on VEXP-enabled hardware with less than 0.1% accuracy loss, removing the need for quantization-aware training.

5. Discussion and Future Work

VEXP demonstrates that massive performance gains in AI workloads do not always require large, fixed-function accelerators. By keeping the logic within the ISA, the system maintains the flexibility to handle diverse code while offloading the most expensive transcendental operations to specialized hardware.

While the current implementation is optimized for Bfloat16, future iterations could explore support for emerging formats like FP8 or Microscaling (MX) formats. Additionally, as exponentiation becomes virtually "free" in terms of cycle count, the division step in Softmax normalization may become the next target for optimization. VEXP proves that targeted, low-cost ISA tweaks can effectively break the bottlenecks of modern Transformer architectures.

Summary updated

BibTeX
@inproceedings{wang2025vexp,
  author = {Run Wang and Gamze Islamoglu and Andrea Belano and Viviane Potocnik and Francesco Conti and Angelo Garofalo and Luca Benini},
  title = {{VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers}},
  booktitle = {ARITH},
  year = {2025},
  doi = {10.1109/arith64983.2025.00016},
}
Figure