This academic paper presents MaxEVA, a framework designed to squeeze every bit of performance out of the AMD/Xilinx Versal AI Engine (AIE) for Matrix Multiplication (MatMul)—the computational heartbeat of deep learning.
TL;DR
MaxEVA is an optimization framework that maximizes throughput and energy efficiency for Matrix Multiplication on Versal AI Engines through analytical modeling and a sophisticated kernel placement strategy. By intelligently managing on-chip data movement and resource mapping, it achieves up to a 2.19× throughput gain and 20.4% higher energy efficiency compared to current state-of-the-art methods.
What Problem Does This Paper Address?
Deep Learning (DL) workloads are expanding at a staggering rate, and traditional FPGAs are evolving into specialized "Adaptive Compute Acceleration Platforms" (ACAPs) like the Versal series to keep up. The Versal AI Engine (AIE) is a massive array of vector processors capable of trillions of operations per second.
However, the AIE architecture is notoriously difficult to program efficiently. Designers face three major hurdles:
- I/O Bottlenecks: There are far fewer interface tiles between the AI Engine and the Programmable Logic (PL) than there are compute cores, creating a data "traffic jam."
- Memory Constraints: Each core has very limited local memory (32KB).
- Routing Congestion: Naive placement of kernels leads to "routing walls" where the hardware cannot physically connect the necessary data paths.
Methods
The authors propose a systematic methodology to map MatMul workloads onto the AIE array:
- Analytical Modeling: They use Integer Programming (IP) to find the optimal tiling sizes ($M, K, N$) and mapping parameters ($X, Y, Z$). This model ensures the design stays within hardware bounds (memory, I/O bandwidth) while maximizing the number of Multiply-Accumulate (MAC) operations per cycle.
- Adder Tree Reduction: Instead of using separate AIE cores for every addition in a reduction tree, MaxEVA maps an entire adder tree to a single AIE core. Because addition is much faster than multiplication, these kernels can run sequentially, saving significant memory by using single buffers instead of double buffers.
- Sophisticated Placement Strategy: The framework uses specific placement patterns (labeled P1 and P2) that exploit "direct memory sharing" between neighboring cores. This allows data to move between tiles without using the slower, more resource-intensive DMA (Direct Memory Access) or AXI-Stream switches.
- Broadcasting: To solve the I/O bottleneck, the framework broadcasts input matrices $A$ and $B$ to multiple cores and performs the final summation (reduction) on the AIE array itself, rather than sending partial results back to the PL.
Datasets
The framework was evaluated using the AMD/Xilinx VC1902 device on a VCK190 evaluation board.
- Hardware Specs: 400 AIE cores operating at 1.25 GHz.
- Benchmarks: The authors compared MaxEVA against CHARM, the previous state-of-the-art framework for AIE MatMul acceleration.
- Tools: Simulations were performed using Vitis 2022.1, the AIE simulator, and the Xilinx Power Estimator (XPE).
Key Insights and Ideas
- Sequential is a Win: One of the most counter-intuitive "aha moments" is that running Add kernels sequentially on one core is better than running them in parallel across many. It saves 50% of the memory buffers needed for those steps and frees up cores for more MatMul operations.
- The "T-Shape" Pattern: To fill the 400-core array without causing routing congestion, the authors discovered that specific "T-shaped" placement patterns are necessary. This accounts for the way AIE rows alternate their memory access directions (East vs. West).
- Bypassing the PL: By performing the reduction of partial sums entirely within the AIE array, the design significantly reduces the number of required output pins (PLIOs), which is often the primary performance bottleneck in ACAP designs.
Key Results
MaxEVA demonstrates a significant leap over existing implementations:
- Throughput (fp32): Achieved 5.44 TFLOPs, a 20.8% improvement over CHARM.
- Throughput (int8): Achieved 77.01 TOPs, a massive 2.19× improvement over the scaled results of CHARM.
- Energy Efficiency: Reached 124.16 GFLOPs/W for fp32, outperforming the state-of-the-art by 20.4%.
- Resource Utilization: The framework successfully utilized 100% of the AIE cores and 99.8% of the available memory in its most dense configurations.
Limitations and Future Work
- Kernel Scope: The current work focuses strictly on Matrix-Matrix multiplication. The authors note that extending this to Matrix-Vector operations is a logical next step.
- Memory Placement: The framework assumes input and output buffers are located in PL BRAM; exploring different memory hierarchies (like DDR or HBM) could be a future direction.
- Automation: While the modeling is analytical, the authors have open-sourced the code to help the community further automate the generation of these optimized kernels for various DL layers.
The code for MaxEVA is available at: https://github.com/enyac-group/MaxEVA
Summary updated