LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition

Ramyad Hadidi, Bahar Asgari, Jiashen Cao, Younmin Bae, Da Eun Shim, Hyojong Kim, Sung Kyu Lim, Michael S. Ryoo, Hyesoon Kim

Abstract

Deep neural networks (DNNs) have inspired new studies in myriad edge applications with robots, autonomous agents, and Internet-of-things (IoT) devices. However, performing inference of DNNs in the edge is still a severe challenge, mainly because of the contradiction between the intensive resource requirements of DNNs and the tight resource availability in several edge domains. Further, as communication is costly, taking advantage of other available edge devices by using data- or model-parallelism methods is not an effective solution. To benefit from available compute resources with low communication overhead, we propose the first DNN parallelization method for reducing the communication overhead in a distributed system. We propose a low-communication parallelization (LCP) method in which models consist of several almost-independent and narrow branches. LCP offers close-to-minimum communication overhead with better distribution and parallelization opportunities while significantly reducing memory footprint and computation compared to data- and model-parallelism methods. We deploy LCP models on three distributed systems: AWS instances, Raspberry Pis, and PYNQ boards. We also evaluate the performance of LCP models on a customized hardware (tailored for low latency) implemented on a small edge FPGA and as a 16mW 0.107mm2 ASIC @7nm chip. LCP models achieve a maximum and average speedups of 56x and 7x, compared to the originals, which could be improved by up to an average speedup of 33x by incorporating common optimizations such as pruning and quantization.

Introduction: The Bottleneck of Edge Inference

Executing modern Deep Neural Networks (DNNs) on edge devices—such as robots, drones, and IoT sensors—presents a significant challenge due to the disparity between model resource requirements and device capabilities. Standard distributed computing solutions often fail in these environments:

  • Cloud Offloading introduces unacceptable latency and privacy risks.
  • Data Parallelism improves throughput but fails to reduce the latency of individual inferences or the memory burden on single nodes.
  • Model Parallelism typically requires frequent inter-layer communication, which becomes a bottleneck in wireless edge networks, leading to the "straggler problem" where the slowest node delays the entire system.

Proposed Method: Low-Communication Parallelization (LCP)

The core innovation of this work is the Low-Communication Parallelization (LCP) method. Unlike traditional monolithic models, LCP architectures consist of multiple narrow, almost-independent branches. These branches only communicate at the input and the final output stages, effectively eliminating the high overhead of intermediate data exchange.

Comparison of LCP with traditional parallelization methods: (a) Data Parallelism, (b) Model Parallelism, (c) Hierarchical-SplitNet, and (d) LCP.

To maintain accuracy while narrowing the model, the authors employ a "fattening" hyperparameter ($F\%$), slightly increasing the width of these independent branches until accuracy is recovered within a target threshold. This approach leverages "feature disjointness," where independent branches learn non-overlapping features more efficiently than a single massive network.

Experimental Evaluation and Results

The researchers evaluated LCP across benchmarks including MNIST, CIFAR-10/100, Flower102, and ImageNet (ILSVRC2012), covering architectures like AlexNet, VGG-16, and ResNet. LCP achieves "superlinear" speedups by drastically reducing communication overhead. For instance, on a cluster of Raspberry Pi 3s, LCP reduced AlexNet's latency from 2.8 seconds to 0.42 seconds using 8 devices.

Top-1 Accuracy vs. Latency on Raspberry Pi 3. LCP (Our Proposed Work) significantly outperforms the single-device Pareto frontier and approaches the ideal zero-communication scenario.

Key performance metrics include:

  • Maximum Speedup: 56x compared to original models.
  • Average Speedup: 7x (increasing to 33x with pruning and quantization).
  • Hardware Efficiency: A custom 7nm ASIC implementation occupied only 0.107mm² and consumed 16mW.

Conclusion and Future Directions

LCP provides a viable path for high-performance DNN inference on resource-constrained edge clusters. While currently requiring manual hyperparameter tuning for branch "fattening," future work aims to automate this process through Neural Architecture Search (NAS). Additionally, expanding LCP to non-CNN architectures like Transformers remains an open area for research.

Summary updated

Abstract

Deep neural networks (DNNs) have inspired new studies in myriad edge applications with robots, autonomous agents, and Internet-of-things (IoT) devices. However, performing inference of DNNs in the edge is still a severe challenge, mainly because of the contradiction between the intensive resource requirements of DNNs and the tight resource availability in several edge domains. Further, as communication is costly, taking advantage of other available edge devices by using data- or model-parallelism methods is not an effective solution. To benefit from available compute resources with low communication overhead, we propose the first DNN parallelization method for reducing the communication overhead in a distributed system. We propose a low-communication parallelization (LCP) method in which models consist of several almost-independent and narrow branches. LCP offers close-to-minimum communication overhead with better distribution and parallelization opportunities while significantly reducing memory footprint and computation compared to data- and model-parallelism methods. We deploy LCP models on three distributed systems: AWS instances, Raspberry Pis, and PYNQ boards. We also evaluate the performance of LCP models on a customized hardware (tailored for low latency) implemented on a small edge FPGA and as a 16mW 0.107mm2 ASIC @7nm chip. LCP models achieve a maximum and average speedups of 56x and 7x, compared to the originals, which could be improved by up to an average speedup of 33x by incorporating common optimizations such as pruning and quantization.
BibTeX
@misc{hadidi2020lcp,
  author = {Ramyad Hadidi and Bahar Asgari and Jiashen Cao and Younmin Bae and Da Eun Shim and Hyojong Kim and Sung Kyu Lim and Michael S. Ryoo and Hyesoon Kim},
  title = {{LCP: A Low-Communication Parallelization Method for Fast Neural Network Inference in Image Recognition}},
  year = {2020},
  eprint = {2003.06464},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.06464},
}
Figure