Understanding the Impact of On-chip Communication on DNN Accelerator Performance

Robert Guirado, Hyoukjun Kwon, Eduard Alarcón, Sergi Abadal, Tushar Krishna

TL;DR

This paper shifts the spotlight from raw computation to the often-overlooked bottleneck in AI hardware: on-chip communication. By analyzing how different dataflow strategies (like those used in Eyeriss or NVDLA) interact with Network-on-Chip (NoC) bandwidth, the authors demonstrate that communication flexibility is just as vital as compute power, paving the way for novel interconnects like Wireless NoCs (WNoCs).

What Problem Does This Paper Address?

While Deep Neural Network (DNN) accelerators have traditionally focused on maximizing Multiply-Accumulate (MAC) operations, the real cost of modern AI is moving data. As we move from cloud-based processing to "edge" devices (like smart home assistants or autonomous cars), hardware is constrained by strict power and area envelopes.

The authors argue that the "dataflow"—the strategy used to map neural network dimensions onto physical processing elements (PEs)—dictates the communication patterns and bandwidth requirements of the chip. If the on-chip network (NoC) cannot keep up with the dataflow, the expensive computational units sit idle. This paper seeks to quantify exactly how NoC bandwidth impacts performance across different dataflows and neural network architectures.

Methods

The researchers employed MAESTRO, an open-source analytical cost model, to simulate the performance of various accelerator configurations.

They characterized five distinct dataflow styles:

  1. No Local Reuse (NLR): A synthetic baseline that keeps no data stationary.
  2. Weight Stationary (WS): Minimizes weight movement (common in many ASICs).
  3. Output Stationary (OS): Minimizes the movement of partial sums (e.g., ShiDiannao).
  4. Row Stationary (RS): A complex hybrid used by the Eyeriss accelerator.
  5. NVDLA Style: A variant of Weight Stationary used in NVIDIA’s Deep Learning Accelerator.

The study modeled an edge-scale accelerator with 256 PEs and 256KB of SRAM, varying the NoC bandwidth from 4 GB/s to 256 GB/s to observe the "roofline" performance limits.

Datasets

The study evaluated these dataflows using two state-of-the-art workloads that represent common computer vision tasks:

  • ResNet50: A standard deep residual network used for image classification.
  • MobileNetV2: A highly optimized, "mobile-first" network that uses depth-wise and point-wise convolutions, which present unique communication challenges compared to standard convolutions.

Key Insights and Ideas

  • The Dataflow-Layer Mismatch: No single dataflow is king. For example, the NVDLA style provides the highest throughput for most layers because it parallelizes over input/output channels. However, in "early" layers (where channel counts are low but image resolution is high), it can underperform compared to styles that parallelize over activations.
  • Bandwidth Sensitivity is Discrete: Throughput doesn't increase linearly with bandwidth. Because data is moved in tiles, performance gains happen in "steps." Increasing bandwidth only helps if it reduces the number of cycles needed to move a full tile of data.
  • The FC Layer Bottleneck: Fully Connected (FC) layers and residual links are bandwidth-hungry. Because they offer limited data reuse, they rely heavily on the NoC to feed the PEs, often requiring significantly more bandwidth than standard convolutional layers to reach peak performance.
  • The Case for WNoC: Traditional wired NoCs (like buses or trees) are often rigid. The authors suggest that Wireless Networks-on-Chip (WNoC) could provide the "on-demand" flexibility needed to support multiple dataflows on a single chip, allowing the topology to virtually reconfigure itself for different layers.

Key Results

  • Throughput Caps: For late layers in ResNet50, throughput is strictly bounded by the number of PEs (256 MACs/cycle). However, reaching this "roofline" requires vastly different bandwidths depending on the dataflow.
  • Bandwidth Requirements: The peak bandwidth requirement for some dataflows (like NLR in FC layers) can exceed 300 GB/s, while more efficient styles like RS or NVDLA can achieve high utilization with much less, provided the layer shape matches their parallelization strategy.
  • NVDLA Efficiency: The NVDLA-style dataflow generally showed the highest "roofline" across various layer types, though it showed lower sensitivity to bandwidth in early layers compared to others.

Limitations and Future Work

The paper is primarily a characterization study based on analytical modeling rather than physical hardware implementation. While it makes a compelling qualitative case for Wireless NoCs, it does not provide a detailed architectural design for one.

The authors note that future work is needed to identify specific "WNoC-aware" dataflows—mapping strategies designed specifically to exploit the broadcast and multi-hop capabilities of wireless interconnects. Additionally, the study focuses on inference; training workloads would likely present even more strenuous communication demands.

Summary updated

Abstract

Deep Neural Networks have flourished at an unprecedented pace in recent years. They have achieved outstanding accuracy in fields such as computer vision, natural language processing, medicine or economics. Specifically, Convolutional Neural Networks (CNN) are particularly suited to object recognition or identification tasks. This, however, comes at a high computational cost, prompting the use of specialized GPU architectures or even ASICs to achieve high speeds and energy efficiency. ASIC accelerators streamline the execution of certain dataflows amenable to CNN computation that imply the constant movement of large amounts of data, thereby turning on-chip communication into a critical function within the accelerator. This paper studies the communication flows within CNN inference accelerators of edge devices, with the aim to justify current and future decisions in the design of the on-chip networks that interconnect their processing elements. Leveraging this analysis, we then qualitatively discuss the potential impact of introducing the novel paradigm of wireless on-chip network in this context.
BibTeX
@inproceedings{guirado2019understanding,
  author = {Robert Guirado and Hyoukjun Kwon and Eduard Alarcón and Sergi Abadal and Tushar Krishna},
  title = {{Understanding the Impact of On-chip Communication on DNN Accelerator Performance}},
  booktitle = {ICECS},
  year = {2019},
  url = {https://openreview.net/forum?id=dlli85oncL},
}

Loading related content...

Figure