Abstract
The increasing computational and memory requirements of Deep Learning (DL) workloads has led to outstanding innovations in hardware architectures. An archetype of such architectures is the novel Versal AI Engine (AIE) by AMD/Xilinx. The AIE comprises multiple programmable processors optimized for vector-based algorithms. An AIE array consisting of 400 processor cores, operating at 1.25 GHz is able to deliver a peak throughput of 8 TFLOPs for 32-bit floating-point (fp32), and 128 TOPs for 8-bit integer (int8) precision. In this work, we propose MaxEVA: a novel framework to efficiently map Matrix Multiplication (MatMul) workloads on Versal AIE devices. Our framework maximizes the performance and energy efficiency of MatMul applications by efficiently exploiting features of the AIE architecture and resolving performance bottlenecks from multiple angles. When demonstrating on the VC1902 device of the VCK190 board, MaxEVA accomplishes up to 5.44 TFLOPs and 77.01 TOPs throughput for fp32 and int8 precisions, respectively. In terms of energy efficiency, MaxEVA attains up to 124.16 GFLOPs/W for fp32, and 1.16 TOPs/W for int8. Our proposed method substantially outperforms the state-of-the-art approach by exhibiting up to 2.19x throughput gain and 20.4% higher energy efficiency. The MaxEVA framework provides notable insights to fill the knowledge gap in effectively designing MatMul-based DL workloads on the new Versal AIE devices.
Introduction
Deep Learning (DL) workloads are expanding at a staggering rate, and traditional FPGAs are evolving into specialized "Adaptive Compute Acceleration Platforms" (ACAPs) like the Versal series to keep up. The Versal AI Engine (AIE) is a massive array of vector processors capable of trillions of operations per second. However, the AIE architecture is notoriously difficult to program efficiently. Designers face significant hurdles including I/O bottlenecks between the AIE and Programmable Logic (PL), limited local memory (32KB per core), and routing congestion.
The MaxEVA Framework
MaxEVA is an optimization framework designed to squeeze maximum performance out of the AIE for Matrix Multiplication (MatMul)—the computational heartbeat of deep learning. It employs a systematic methodology to map MatMul workloads onto the AIE array through analytical modeling and hardware-aware strategies.
Analytical Modeling and Broadcasting
The framework uses Integer Programming (IP) to find the optimal tiling sizes ($M, K, N$) and mapping parameters ($X, Y, Z$). This model ensures the design stays within hardware bounds while maximizing the number of Multiply-Accumulate (MAC) operations per cycle. To solve the I/O bottleneck, MaxEVA broadcasts input matrices $A$ and $B$ to multiple cores and performs the final summation (reduction) on the AIE array itself, rather than sending partial results back to the PL.
Adder Tree Reduction
Instead of using separate AIE cores for every addition in a reduction tree, MaxEVA maps an entire adder tree to a single AIE core. Because addition is much faster than multiplication, these kernels can run sequentially. This "sequential win" saves 50% of the memory buffers needed for those steps, freeing up cores for more MatMul operations.
Sophisticated Placement Strategy
The framework uses specific placement patterns that exploit "direct memory sharing" between neighboring cores. This allows data to move between tiles without using the slower, more resource-intensive DMA or AXI-Stream switches. To fill the 400-core array without causing routing congestion, MaxEVA utilizes "T-shaped" placement patterns that account for the way AIE rows alternate their memory access directions (East vs. West).
Experimental Evaluation
The framework was evaluated using the AMD/Xilinx VC1902 device on a VCK190 evaluation board, featuring 400 AIE cores operating at 1.25 GHz. MaxEVA was compared against CHARM, the previous state-of-the-art framework for AIE MatMul acceleration.
MaxEVA demonstrates a significant leap over existing implementations:
- Throughput (fp32): Achieved 5.44 TFLOPs, a 20.8% improvement over CHARM.
- Throughput (int8): Achieved 77.01 TOPs, a 2.19× improvement over the scaled results of CHARM.
- Energy Efficiency: Reached 124.16 GFLOPs/W for fp32, outperforming the state-of-the-art by 20.4%.
- Resource Utilization: The framework successfully utilized 100% of the AIE cores and 99.8% of the available memory in its most dense configurations.
Conclusion
MaxEVA provides a robust framework for mapping MatMul-based DL workloads on the new Versal AIE devices. By intelligently managing on-chip data movement and resource mapping, it achieves substantial gains in both throughput and energy efficiency. Future work will focus on extending the framework to Matrix-Vector operations and exploring diverse memory hierarchies such as HBM.
The code for MaxEVA is available at: https://github.com/enyac-group/MaxEVA
Summary updated