Abstract
Deep Neural Networks (DNNs) have flourished at an unprecedented pace in recent years, achieving outstanding accuracy in fields such as computer vision and natural language processing. Specifically, Convolutional Neural Networks (CNNs) are particularly suited to object recognition tasks. This, however, comes at a high computational cost, prompting the use of specialized ASIC accelerators. These accelerators streamline execution via specific dataflows that imply the constant movement of large amounts of data, turning on-chip communication into a critical function. This paper studies communication flows within CNN inference accelerators for edge devices to justify design decisions for the on-chip networks (NoCs) that interconnect processing elements (PEs). Leveraging this analysis, we qualitatively discuss the potential impact of introducing wireless on-chip networks (WNoCs) to provide the flexibility required by modern neural architectures.
Introduction
While DNN accelerator research has traditionally focused on maximizing Multiply-Accumulate (MAC) operations, the primary bottleneck in modern AI hardware is often data movement. As processing shifts toward "edge" devices with strict power and area constraints, the efficiency of the on-chip network becomes paramount. The "dataflow"—the strategy used to map high-dimensional neural network operations onto physical hardware—dictates the communication patterns and bandwidth requirements of the chip. If the NoC cannot keep up with the dataflow, expensive computational units sit idle.
Methodology
To quantify the impact of NoC bandwidth, researchers employed MAESTRO, an open-source analytical cost model. They simulated an edge-scale accelerator featuring 256 PEs and 256KB of SRAM, varying the NoC bandwidth from 4 GB/s to 256 GB/s. The study characterized five distinct dataflow styles:
- No Local Reuse (NLR): A baseline with no data stationarity.
- Weight Stationary (WS): Minimizes weight movement.
- Output Stationary (OS): Minimizes partial sum movement (e.g., ShiDiannao).
- Row Stationary (RS): A hybrid approach used in the Eyeriss accelerator.
- NVDLA Style: A variant of Weight Stationary used in NVIDIA’s Deep Learning Accelerator.
These dataflows were evaluated using ResNet50 and MobileNetV2, representing standard and mobile-optimized computer vision workloads.
Results and Discussion
Bandwidth Sensitivity and the "Step" Effect
A key insight from the simulation is that throughput does not increase linearly with bandwidth. Because data is moved in discrete tiles, performance gains occur in "steps." Increasing bandwidth only improves performance if it reduces the total number of cycles needed to move a full tile of data to the PEs.
The Dataflow-Layer Mismatch
No single dataflow is optimal for every layer of a network. For instance, the NVDLA style generally provides high throughput by parallelizing over input/output channels. However, in "early" layers where image resolution is high but channel counts are low, it can underperform compared to styles that parallelize over activations. Conversely, "late" layers and Fully Connected (FC) layers often act as bandwidth bottlenecks. Because these layers offer limited data reuse, they rely heavily on the NoC to feed the PEs, sometimes requiring over 300 GB/s to reach peak utilization.
The Case for Wireless NoCs
Traditional wired NoCs (like buses or trees) are often too rigid to handle the varying demands of different layers efficiently. The authors suggest that Wireless Networks-on-Chip (WNoC) could provide the "on-demand" flexibility needed. By allowing the topology to virtually reconfigure itself for different layers, a WNoC could support multiple dataflows on a single chip more effectively than static wired interconnects.
Conclusion
This characterization highlights that communication flexibility is just as vital as raw compute power in DNN accelerators. While efficient dataflows like Row Stationary and NVDLA-style can reach performance rooflines with moderate bandwidth, the diversity of layer shapes in modern networks necessitates more adaptable interconnects. Future work should focus on identifying "WNoC-aware" dataflows designed specifically to exploit the broadcast and multi-hop capabilities of wireless on-chip communication.
Summary updated