TL;DR
This paper shifts the spotlight from raw computation to the often-overlooked bottleneck in AI hardware: on-chip communication. By analyzing how different dataflow strategies (like those used in Eyeriss or NVDLA) interact with Network-on-Chip (NoC) bandwidth, the authors demonstrate that communication flexibility is just as vital as compute power, paving the way for novel interconnects like Wireless NoCs (WNoCs).
What Problem Does This Paper Address?
While Deep Neural Network (DNN) accelerators have traditionally focused on maximizing Multiply-Accumulate (MAC) operations, the real cost of modern AI is moving data. As we move from cloud-based processing to "edge" devices (like smart home assistants or autonomous cars), hardware is constrained by strict power and area envelopes.
The authors argue that the "dataflow"—the strategy used to map neural network dimensions onto physical processing elements (PEs)—dictates the communication patterns and bandwidth requirements of the chip. If the on-chip network (NoC) cannot keep up with the dataflow, the expensive computational units sit idle. This paper seeks to quantify exactly how NoC bandwidth impacts performance across different dataflows and neural network architectures.
Methods
The researchers employed MAESTRO, an open-source analytical cost model, to simulate the performance of various accelerator configurations.
They characterized five distinct dataflow styles:
- No Local Reuse (NLR): A synthetic baseline that keeps no data stationary.
- Weight Stationary (WS): Minimizes weight movement (common in many ASICs).
- Output Stationary (OS): Minimizes the movement of partial sums (e.g., ShiDiannao).
- Row Stationary (RS): A complex hybrid used by the Eyeriss accelerator.
- NVDLA Style: A variant of Weight Stationary used in NVIDIA’s Deep Learning Accelerator.
The study modeled an edge-scale accelerator with 256 PEs and 256KB of SRAM, varying the NoC bandwidth from 4 GB/s to 256 GB/s to observe the "roofline" performance limits.
Datasets
The study evaluated these dataflows using two state-of-the-art workloads that represent common computer vision tasks:
- ResNet50: A standard deep residual network used for image classification.
- MobileNetV2: A highly optimized, "mobile-first" network that uses depth-wise and point-wise convolutions, which present unique communication challenges compared to standard convolutions.
Key Insights and Ideas
- The Dataflow-Layer Mismatch: No single dataflow is king. For example, the NVDLA style provides the highest throughput for most layers because it parallelizes over input/output channels. However, in "early" layers (where channel counts are low but image resolution is high), it can underperform compared to styles that parallelize over activations.
- Bandwidth Sensitivity is Discrete: Throughput doesn't increase linearly with bandwidth. Because data is moved in tiles, performance gains happen in "steps." Increasing bandwidth only helps if it reduces the number of cycles needed to move a full tile of data.
- The FC Layer Bottleneck: Fully Connected (FC) layers and residual links are bandwidth-hungry. Because they offer limited data reuse, they rely heavily on the NoC to feed the PEs, often requiring significantly more bandwidth than standard convolutional layers to reach peak performance.
- The Case for WNoC: Traditional wired NoCs (like buses or trees) are often rigid. The authors suggest that Wireless Networks-on-Chip (WNoC) could provide the "on-demand" flexibility needed to support multiple dataflows on a single chip, allowing the topology to virtually reconfigure itself for different layers.
Key Results
- Throughput Caps: For late layers in ResNet50, throughput is strictly bounded by the number of PEs (256 MACs/cycle). However, reaching this "roofline" requires vastly different bandwidths depending on the dataflow.
- Bandwidth Requirements: The peak bandwidth requirement for some dataflows (like NLR in FC layers) can exceed 300 GB/s, while more efficient styles like RS or NVDLA can achieve high utilization with much less, provided the layer shape matches their parallelization strategy.
- NVDLA Efficiency: The NVDLA-style dataflow generally showed the highest "roofline" across various layer types, though it showed lower sensitivity to bandwidth in early layers compared to others.
Limitations and Future Work
The paper is primarily a characterization study based on analytical modeling rather than physical hardware implementation. While it makes a compelling qualitative case for Wireless NoCs, it does not provide a detailed architectural design for one.
The authors note that future work is needed to identify specific "WNoC-aware" dataflows—mapping strategies designed specifically to exploit the broadcast and multi-hop capabilities of wireless interconnects. Additionally, the study focuses on inference; training workloads would likely present even more strenuous communication demands.
Summary updated