Striking the Balance: GEMM Performance Optimization Across Generations of Ryzen AI NPUs

Endri Taka, Andre Roesti, Joseph Melber, Pranathi Vasireddy, Kristof Denolf, Diana MarculescuFPGA 2025 · arXiv · OpenReview

TL;DR

This paper presents a systematic methodology to optimize General Matrix Multiplication (GEMM) performance on AMD's Ryzen AI NPUs, specifically across the XDNA and XDNA2 generations. By carefully balancing compute and memory bandwidth through analytical modeling and hardware profiling, the authors achieve state-of-the-art GEMM throughput, offering significant insights for deep learning acceleration on edge devices.

What Problem Does This Paper Address?

Modern deep learning (DL) applications demand immense computational power, driving the development of specialized hardware accelerators like AMD's Ryzen AI NPUs. A core operation in DL is General Matrix Multiplication (GEMM). Optimizing GEMM for these NPUs is crucial for unlocking their full potential. However, the unique architecture of NPUs, particularly the absence of FPGA fabric found in previous AMD Versal devices, presents distinct challenges. This paper addresses the need for a unified, systematic approach to maximize GEMM performance on these new NPU generations, considering the intricate interplay between on-chip computation and off-chip memory bandwidth.

Methods

The authors propose a multi-level tiling strategy and a sophisticated data movement architecture to optimize GEMM. Key techniques include:

  • Systematic Methodology: A unified framework combining analytical modeling and hardware profiling to identify the optimal "balanced point" where compute and memory bandwidth are equally utilized, maximizing performance.
  • On-the-Fly Tensor Transformations: Leveraging the NPU's multi-dimensional DMA addressing capabilities to transform data layouts directly during transfers, allowing matrices to remain in standard row/column-major order in DRAM without explicit pre-tiling. This simplifies integration with existing DL libraries.
  • Output-Stationary Mapping: A GEMM mapping strategy where output tiles remain in L1 memory for accumulation, reducing memory traffic and freeing up L1 space for input tiles, enabling larger tile sizes and better performance.
  • Double-Buffering for Inputs: Employing double-buffering for input matrices A and B to overlap computation with data transfers, further enhancing efficiency.
  • Analytical Modeling: Developing equations to model compute time and memory access time, and formulating an integer programming problem to find optimal kernel parameters (mct, kct, nct) that balance these two factors.
  • NPU Array Optimization: Extending the analytical model to consider system-level DRAM accesses and NPU array parallelism, aiming to find the optimal balanced point for the entire NPU.
  • Overlapping BD Reconfiguration: A novel procedure to overlap DMA data transfers with Buffer Descriptor (BD) reconfiguration, preventing performance degradation when complex data movement patterns require more than the available BDs.

Datasets

Not discussed in the paper. The evaluation uses synthetic GEMM workloads of various sizes.

Key Insights and Ideas

  • Compute vs. Memory Bandwidth is Key: The paper strongly emphasizes the inverse relationship between compute time and memory access time. Maximizing GEMM performance isn't about pushing raw compute to its limit, but finding the sweet spot where both are balanced.
  • DRAM Layout Matters Significantly: Storing matrix B in column-major order consistently yields higher performance than row-major, especially on XDNA2. This is attributed to better contiguous DRAM access, which is critical for maximizing memory bandwidth utilization.
  • On-the-Fly Transformations Simplify Integration: The ability to perform data layout transformations directly within the DMA engine is a game-changer. It allows matrices to be stored in standard formats in DRAM, making it easier to integrate optimized GEMM kernels with existing DL frameworks.
  • Output-Stationary Mapping is Efficient: Keeping output tiles in L1 memory for accumulation is more efficient than double-buffering output, as it frees up valuable L1 cache for input data, enabling larger, more performant tiles.
  • BD Reconfiguration Overlap is Crucial: The proposed method to overlap DMA transfers with BD reconfiguration is essential for achieving high performance, preventing significant bottlenecks that would otherwise occur.

Key Results

The paper reports impressive GEMM performance figures:

  • XDNA: Up to 6.76 TOPS (int8) and 3.14 TOPS (bf16) for 8-bit integer and brain floating-point precision, respectively.
  • XDNA2: Up to 38.05 TOPS (int8) and 14.71 TOPS (bf16) for 8-bit integer and brain floating-point precision, respectively.
  • The optimized kernels achieve significantly higher performance compared to previous non-optimized GEMM implementations on XDNA (e.g., up to 34% higher for int8).
  • The methodology successfully identifies optimal balanced kernels that balance compute and memory, leading to substantial performance gains over kernels optimized solely for compute.
  • The system-level GEMM performance sweeps (roofline plots) demonstrate that performance scales with arithmetic intensity and becomes memory-bound at lower intensities.

Limitations and Future Work

  • bfp16 Precision: XDNA2 has hardware support for bfp16 precision, which introduces additional challenges for data layout transformations. This is left for future work.
  • GEMV Optimization: The proposed methodology could be extended to optimize other GEMM-like operations such as general matrix-vector multiplication (GEMV).
  • On-the-Fly Zero-Padding: While the NPU supports on-the-fly zero-padding, this feature was not fully exploited in this work and is suggested for future exploration.
  • Reconfiguration Latency: While the current approach minimizes reconfiguration overhead, switching between vastly different GEMM sizes can still incur noticeable latency, which is a general challenge for dynamic workloads.

Summary updated

Abstract

The high computational and memory demands of modern deep learning (DL) workloads have led to the development of specialized hardware devices from cloud to edge, such as AMD's Ryzen AI XDNA NPUs. Optimizing general matrix multiplication (GEMM) algorithms for these architectures is critical for improving DL workload performance. To this end, this paper presents a common systematic methodology to optimize GEMM workloads across the two current NPU generations, namely XDNA and XDNA2. Our implementations exploit the unique architectural features of AMD's NPUs and address key performance bottlenecks at the system level. End-to-end performance evaluation across various GEMM sizes demonstrates state-of-the-art throughput of up to 6.76 TOPS (XDNA) and 38.05 TOPS (XDNA2) for 8-bit integer (int8) precision. Similarly, for brain floating-point (bf16) precision, our GEMM implementations attain up to 3.14 TOPS (XDNA) and 14.71 TOPS (XDNA2). This work provides significant insights into key performance aspects of optimizing GEMM workloads on Ryzen AI NPUs.
BibTeX
@inproceedings{taka2025striking,
  author = {Endri Taka and Andre Roesti and Joseph Melber and Pranathi Vasireddy and Kristof Denolf and Diana Marculescu},
  title = {{Striking the Balance: GEMM Performance Optimization Across Generations of Ryzen AI NPUs}},
  booktitle = {FPGA},
  year = {2025},
  url = {https://openreview.net/forum?id=Zw7ORCfSxe},
}
Figure