TL;DR
This paper introduces a reconfigurable matrix engine that bridges the gap between low-latency signal processing and high-throughput AI inference. By using low-cost multiplexers to transform a standard systolic array into a tensor-style engine on the fly, the design achieves up to a 7.5× latency reduction for wireless kernels and a 1.65× speedup for neural networks with minimal hardware overhead.
What Problem Does This Paper Address?
Modern mobile devices are increasingly tasked with two very different types of heavy lifting: physical-layer (PHY) signal processing (like 5G/6G MIMO detection) and Deep Neural Network (DNN) inference. Both rely heavily on matrix multiplication, but their requirements are diametrically opposed.
- Signal Processing typically involves small, serialized matrix operations where low latency is critical.
- DNN Inference involves massive matrices where high throughput and energy efficiency (via data reuse) are the priorities.
Traditionally, hardware designers choose between two architectures:
- Systolic Arrays (SA): Excellent for large matrices due to high data reuse, but they suffer from "fill-and-drain" delays that kill performance on small matrices.
- Tensor Units: Fast for small operations because they perform reductions in parallel, but they require massive memory bandwidth and offer poor data reuse for large workloads.
The author argues that we shouldn't have to choose one or the other, especially in resource-constrained mobile environments.
Methods
The core contribution is a reconfigurable matrix engine that can operate as either a 2D Systolic Array or a "3D" Tensor engine.
Unlike previous "reshaping" techniques that just change the dimensions of the array, this approach reconfigures the reduction dimension itself. The author uses a set of three routing fabrics (mostly 2:1 multiplexers) to repurpose the existing Multiply-Accumulate (MAC) units and registers within a 64×64 systolic array.
- Systolic Mode: Operates as a standard output-stationary array, propagating operands across rows and columns.
- Tensor Mode: Reorganizes the 4,096 MACs into sixteen 16×16 planes. The existing accumulators are interconnected to form a spatial reduction tree, allowing the engine to complete small kernels in far fewer cycles.
Crucially, the design keeps the peripheral memory system and interfaces identical for both modes. This "limited bandwidth" tensor mode ensures that the hardware overhead remains low while still providing significant latency gains.
Datasets
The engine was evaluated using the SCALE-Sim simulator across two distinct workload domains:
- Wireless PHY Kernels: Common telecom operations including Gram matrix calculation, Matched filtering, MMSE equalization, and Neumann series multiplication.
- DNN Models: A variety of popular architectures including AlexNet, ResNet-50, GoogLeNet, MobileNet, YOLO-tiny, and DeepSpeech2.
For hardware validation, the design was verified on a Zynq 702 FPGA and synthesized using the ASAP 7nm FinFET library.
Key Insights and Ideas
- The "Aha" Moment: The realization that the hardware resources for temporal accumulation in a systolic array can be rewired into a spatial reduction tree for a tensor engine with very little extra logic.
- Per-Operation Reconfiguration: The mode is not fixed at boot; it can be selected by a compiler or scheduler on a per-operation basis, allowing a single chip to handle a complex pipeline of mixed matrix sizes efficiently.
- Bandwidth Mitigation for Complex Numbers: For wireless PHY (which uses complex numbers), the author suggests generating real matrix components locally from complex inputs. This effectively halves the number of cycles needed to load new matrices in the bandwidth-limited tensor mode.
Key Results
- Latency Gains: For small wireless PHY kernels, the tensor mode achieved 2.15× to 7.52× lower latency compared to a vanilla systolic array.
- AI Speedup: On DNN workloads, the reconfigurable engine provided up to a 1.65× speedup (specifically for MobileNet), while maintaining the memory efficiency of an SA for large-batch models like ResNet-50.
- Low Overhead: Synthesis results show that the reconfigurability adds only 13.6% to the area and 8% to the power consumption compared to a standard systolic array.
- Energy Efficiency: The design retains the systolic array's memory access advantage for large workloads, consuming roughly 2× less energy than a pure tensor engine for ResNet-50 inference.
Limitations and Future Work
- Bandwidth Bottleneck: In tensor mode, the engine is still limited by the systolic array's original memory interface. While the paper shows this is often acceptable, it prevents the engine from reaching the absolute peak performance of an unrestricted tensor core.
- Mode Selection Policy: The current selection policy is relatively simple (choosing the mode with fewer cycles). Future work could involve more sophisticated scheduling that balances performance, power, and thermal constraints.
- Scope: The evaluation focuses primarily on matrix multiplication. While this is the dominant operation, other layers (like normalization or activation) were not the focus of this specific architectural optimization.
Summary updated