MXDOTP: A RISC-V ISA Extension for Enabling Microscaling (MX) Floating-Point Dot Products

Gamze Islamoglu, Luca Bertaccini, Arpan Suravi Prasad, Francesco Conti, Angelo Garofalo, Luca Benini

Abstract

As Large Language Models (LLMs) and AI workloads demand increasingly efficient memory and compute utilization, low-precision formats have become essential. However, standard 8-bit floating-point formats often struggle to balance dynamic range and precision. Microscaling (MX) formats address this through block-based scaling, but their implementation in software on general-purpose processors incurs significant overhead. This paper presents MXDOTP, a specialized RISC-V ISA extension and hardware unit designed to natively accelerate MX dot products. Integrated into the Snitch RISC-V core and fabricated in a 12nm FinFET process, the MXDOTP unit achieves a 25x speedup and 12.5x better energy efficiency compared to software-only implementations, with a minimal area overhead of 5.1%.

Introduction

The rapid scaling of AI models has necessitated a shift toward low-bitwidth data formats to mitigate memory bandwidth bottlenecks. While 8-bit floating-point (FP8) formats are increasingly common, they present a rigid trade-off: allocating bits to the exponent increases dynamic range but sacrifices precision, while allocating them to the mantissa does the opposite.

Microscaling (MX) formats, recently standardized by the Open Compute Project (OCP), offer a solution through "block scaling." In this approach, a group of elements (a block) shares a single wide exponent (the scale), while individual elements maintain narrow exponents. While mathematically efficient, this format introduces a "format conversion tax" on standard hardware. Traditional processors are not designed to simultaneously fetch data, fetch scales, and perform fused arithmetic, leading to significant performance degradation when MX operations are emulated in software.

The MXDOTP Architecture

To overcome these limitations, we developed the MXDOTP (Microscaling Dot Product) unit, a dedicated hardware accelerator integrated into the Snitch RISC-V core. The architecture focuses on three primary innovations:

1. The mxdotp Instruction

We introduced a novel five-operand instruction that fuses the scaling and dot-product accumulation into a single pipeline stage. This instruction processes blocks of eight 8-bit operands packed into 64-bit inputs, significantly reducing the instruction count required for matrix operations.

2. Stream Semantic Registers (SSRs)

Standard RISC-V register files typically feature only three read ports, which is insufficient for the multi-operand requirements of mxdotp (which requires data blocks plus their corresponding scales). To avoid the power and area costs of a multi-ported register file, we utilized Stream Semantic Registers (SSRs). SSRs allow data and scales to be streamed directly from memory into the Floating-Point Unit (FPU), bypassing the register file and ensuring the pipeline remains saturated.

3. High-Precision Accumulation

To maintain numerical integrity during the dot product, the unit utilizes an internal FP9 intermediate format and a 95-bit fixed-point accumulator. This architecture ensures that the sum of products is calculated exactly before the final conversion back to FP32, preventing the precision loss typically associated with low-bitwidth arithmetic.

Experimental Results

The design was implemented in a 12nm FinFET process within an 8-core cluster sharing a scratchpad memory. Evaluation was conducted using Matrix Multiplication (MM) kernels and the DeiT-Tiny (Vision Transformer) model, with weights and activations quantized to MXFP8 using Microsoft’s MX PyTorch Emulation Library.

Performance and Efficiency

The MXDOTP extension demonstrated substantial gains over both standard FP32 and software-emulated FP8 approaches:

  • Throughput: The MXFP8 kernel reached 102 GFLOPS on the cluster, a 3.1x to 3.4x speedup over standard FP32 implementations.
  • Energy Efficiency: The system achieved 356 GFLOPS/W at 0.8V, representing a 12.5x improvement in energy efficiency compared to software-based MX math.
  • Hardware Utilization: By leveraging SSRs and the Floating-point Repetition (FREP) extension, the hardware maintained 80% utilization, effectively eliminating stalls related to data movement.

Area Overhead

Despite the performance improvements, the integration of the MXDOTP unit is highly efficient. The unit increased the total cluster area by only 5.1%, making it a low-cost addition for AI-centric RISC-V hardware.

Discussion and Future Work

Our results highlight that native hardware support is essential for realizing the benefits of MX formats; without it, the overhead of bit-shuffling and type conversion negates the efficiency of the smaller format.

Currently, the implementation is optimized for MXFP8 (E5M2 and E4M3 sub-formats) and supports the roundTiesToEven (RNE) rounding mode. Future iterations could expand this support to include MXINT8 and MXFP4 formats, as well as additional rounding modes for scientific computing. Furthermore, developing compiler support (e.g., LLVM/GCC) will be critical for enabling high-level language support and broader adoption of the MXDOTP ISA extension.

Conclusion

MXDOTP provides a high-performance, energy-efficient pathway for integrating OCP-standard Microscaling formats into the RISC-V ecosystem. By fusing scaling into the dot-product pipeline and utilizing streaming registers, the architecture achieves significant throughput gains with minimal area impact, offering a scalable solution for next-generation AI accelerators.

Summary updated

BibTeX
@inproceedings{islamoglu2025mxdotp,
  author = {Gamze Islamoglu and Luca Bertaccini and Arpan Suravi Prasad and Francesco Conti and Angelo Garofalo and Luca Benini},
  title = {{MXDOTP: A RISC-V ISA Extension for Enabling Microscaling (MX) Floating-Point Dot Products}},
  booktitle = {ASAP},
  year = {2025},
  doi = {10.1109/asap65064.2025.00021},
}
Figure