Understanding the Impact of On-chip Communication on DNN Accelerator Performance

Robert Guirado, Hyoukjun Kwon, Eduard Alarcón, Sergi Abadal, Tushar Krishna

Abstract

Deep Neural Networks (DNNs) have flourished at an unprecedented pace in recent years, achieving outstanding accuracy in fields such as computer vision and natural language processing. Specifically, Convolutional Neural Networks (CNNs) are particularly suited to object recognition tasks. This, however, comes at a high computational cost, prompting the use of specialized ASIC accelerators. These accelerators streamline execution via specific dataflows that imply the constant movement of large amounts of data, turning on-chip communication into a critical function. This paper studies communication flows within CNN inference accelerators for edge devices to justify design decisions for the on-chip networks (NoCs) that interconnect processing elements (PEs). Leveraging this analysis, we qualitatively discuss the potential impact of introducing wireless on-chip networks (WNoCs) to provide the flexibility required by modern neural architectures.

Introduction

While DNN accelerator research has traditionally focused on maximizing Multiply-Accumulate (MAC) operations, the primary bottleneck in modern AI hardware is often data movement. As processing shifts toward "edge" devices with strict power and area constraints, the efficiency of the on-chip network becomes paramount. The "dataflow"—the strategy used to map high-dimensional neural network operations onto physical hardware—dictates the communication patterns and bandwidth requirements of the chip. If the NoC cannot keep up with the dataflow, expensive computational units sit idle.

Mapping of CNN dimensions and dataflow execution loops

Methodology

To quantify the impact of NoC bandwidth, researchers employed MAESTRO, an open-source analytical cost model. They simulated an edge-scale accelerator featuring 256 PEs and 256KB of SRAM, varying the NoC bandwidth from 4 GB/s to 256 GB/s. The study characterized five distinct dataflow styles:

  1. No Local Reuse (NLR): A baseline with no data stationarity.
  2. Weight Stationary (WS): Minimizes weight movement.
  3. Output Stationary (OS): Minimizes partial sum movement (e.g., ShiDiannao).
  4. Row Stationary (RS): A hybrid approach used in the Eyeriss accelerator.
  5. NVDLA Style: A variant of Weight Stationary used in NVIDIA’s Deep Learning Accelerator.

These dataflows were evaluated using ResNet50 and MobileNetV2, representing standard and mobile-optimized computer vision workloads.

Results and Discussion

Bandwidth Sensitivity and the "Step" Effect

A key insight from the simulation is that throughput does not increase linearly with bandwidth. Because data is moved in discrete tiles, performance gains occur in "steps." Increasing bandwidth only improves performance if it reduces the total number of cycles needed to move a full tile of data to the PEs.

Throughput vs. NoC bandwidth for late layers, illustrating discrete performance steps

The Dataflow-Layer Mismatch

No single dataflow is optimal for every layer of a network. For instance, the NVDLA style generally provides high throughput by parallelizing over input/output channels. However, in "early" layers where image resolution is high but channel counts are low, it can underperform compared to styles that parallelize over activations. Conversely, "late" layers and Fully Connected (FC) layers often act as bandwidth bottlenecks. Because these layers offer limited data reuse, they rely heavily on the NoC to feed the PEs, sometimes requiring over 300 GB/s to reach peak utilization.

The Case for Wireless NoCs

Traditional wired NoCs (like buses or trees) are often too rigid to handle the varying demands of different layers efficiently. The authors suggest that Wireless Networks-on-Chip (WNoC) could provide the "on-demand" flexibility needed. By allowing the topology to virtually reconfigure itself for different layers, a WNoC could support multiple dataflows on a single chip more effectively than static wired interconnects.

Conclusion

This characterization highlights that communication flexibility is just as vital as raw compute power in DNN accelerators. While efficient dataflows like Row Stationary and NVDLA-style can reach performance rooflines with moderate bandwidth, the diversity of layer shapes in modern networks necessitates more adaptable interconnects. Future work should focus on identifying "WNoC-aware" dataflows designed specifically to exploit the broadcast and multi-hop capabilities of wireless on-chip communication.

Summary updated

Abstract

Deep Neural Networks have flourished at an unprecedented pace in recent years. They have achieved outstanding accuracy in fields such as computer vision, natural language processing, medicine or economics. Specifically, Convolutional Neural Networks (CNN) are particularly suited to object recognition or identification tasks. This, however, comes at a high computational cost, prompting the use of specialized GPU architectures or even ASICs to achieve high speeds and energy efficiency. ASIC accelerators streamline the execution of certain dataflows amenable to CNN computation that imply the constant movement of large amounts of data, thereby turning on-chip communication into a critical function within the accelerator. This paper studies the communication flows within CNN inference accelerators of edge devices, with the aim to justify current and future decisions in the design of the on-chip networks that interconnect their processing elements. Leveraging this analysis, we then qualitatively discuss the potential impact of introducing the novel paradigm of wireless on-chip network in this context.
BibTeX
@inproceedings{guirado2019understanding,
  author = {Robert Guirado and Hyoukjun Kwon and Eduard Alarcón and Sergi Abadal and Tushar Krishna},
  title = {{Understanding the Impact of On-chip Communication on DNN Accelerator Performance}},
  booktitle = {ICECS},
  year = {2019},
  url = {https://openreview.net/forum?id=dlli85oncL},
}
Figure