DISCO: Distributed Inference with Sparse Communications

Minghai Qin, Chao Sun, Jaco Hofmann, Dejan Vucinic

Abstract

Distributed inference of Deep Neural Networks (DNNs) on low-power edge devices is often limited by a "communication wall," where the time spent transferring data between nodes far exceeds computation time. DISCO (Distributed Inference with Sparse Communications) addresses this bottleneck by employing within-layer model parallelism combined with sparse feature map exchange. By framing feature selection as a structured weight pruning problem, DISCO reduces communication overhead by up to 90% while maintaining high accuracy across various computer vision tasks, including classification, detection, and segmentation.

Introduction

Modern DNNs are increasingly massive, often requiring memory resources that far exceed the capabilities of individual edge devices like microcontrollers or smart cameras. While distributing the workload across multiple devices is a natural solution, standard model parallelism requires every node to receive nearly all feature data from the previous layer to compute its portion of the next. On constrained edge networks (e.g., Wi-Fi or Bluetooth), this results in a communication bottleneck where latency is dominated by data transmission rather than processing.

Methodology: Sparse Within-Layer Parallelism

DISCO introduces a novel approach to within-layer model parallelism that prioritizes communication efficiency. Instead of the traditional "dense" exchange where all nodes share all calculated features, DISCO transmits only a carefully selected subset of feature maps.

Comparison of distributed inference architectures and the DISCO sparse communication mechanism.

The Pruning Equivalence

The core technical innovation of DISCO is the realization that selecting which features to transmit is mathematically equivalent to structured weight pruning. If a node does not receive a specific feature from a peer, the weights associated with that feature in the node's local calculation are effectively zeroed. The DISCO pipeline follows three main steps:

  1. Identify Importance: Use the $L_1$ norm to identify the least important weights corresponding to cross-node communication.
  2. Prune and Fine-tune: Prune these weights to enforce sparse communication and fine-tune the remaining parameters to recover model accuracy.
  3. Latency Hiding: Leverage the reduced data volume to pipeline communication, sending required data for subsequent layers while current computations are still in progress.

Experimental Results

The researchers evaluated DISCO using diverse datasets, including ImageNet (classification), COCO2017 (object detection), PASCAL VOC2012 (segmentation), and DIV2K (super-resolution).

Performance comparison of DISCO (Model Parl.) vs. Pipeline Parallelism and Single Node inference.

Key Findings

  • Communication Efficiency: DISCO achieved a 90% reduction in communication for SSD object detection and a 95% reduction for DeepLabV3+ semantic segmentation with negligible performance loss.
  • Latency Advantages: Unlike "Pipeline Parallelism," which processes layers sequentially across nodes and increases total latency, DISCO's within-layer approach parallelizes the work of every layer. As shown in the performance analysis, DISCO maintains low inference latency even as the number of nodes (N) increases, while simultaneously improving accuracy.
  • The "Goldilocks" Zone: The study found that adding even a tiny amount of communication (e.g., 1%) back into a completely split model can significantly boost accuracy, proving that sparse communication is superior to both fully dense and fully independent architectures.

Conclusion

DISCO effectively breaks the communication bottleneck for distributed AI at the edge. By treating communication as a pruning task, it enables large-scale DNN inference on resource-constrained hardware without the prohibitive latency costs of traditional distribution methods. Future work may explore more advanced pruning techniques or heterogeneous network environments to further optimize these distributed systems.

Summary updated

Abstract

Deep neural networks (DNNs) have great potential to solve many real-world problems, but they usually require an extensive amount of computation and memory. It is of great difficulty to deploy a large DNN model to a single resource-limited device with small memory capacity. Distributed computing is a common approach to reduce single-node memory consumption and to accelerate the inference of DNN models. In this paper, we explore the "within-layer model parallelism", which distributes the inference of each layer into multiple nodes. In this way, the memory requirement can be distributed to many nodes, making it possible to use several edge devices to infer a large DNN model. Due to the dependency within each layer, data communications between nodes during this parallel inference can be a bottleneck when the communication bandwidth is limited. We propose a framework to train DNN models for Distributed Inference with Sparse Communications (DISCO). We convert the problem of selecting which subset of data to transmit between nodes into a model optimization problem, and derive models with both computation and communication reduction when each layer is inferred on multiple nodes. We show the benefit of the DISCO framework on a variety of CV tasks such as image classification, object detection, semantic segmentation, and image super resolution. The corresponding models include important DNN building blocks such as convolutions and transformers. For example, each layer of a ResNet-50 model can be distributively inferred across two nodes with five times less data communications, almost half overall computations and half memory requirement for a single node, and achieve comparable accuracy to the original ResNet-50 model. This also results in 4.7 times overall inference speedup.
BibTeX
@inproceedings{qin2024disco,
  author = {Minghai Qin and Chao Sun and Jaco Hofmann and Dejan Vucinic},
  title = {{DISCO: Distributed Inference with Sparse Communications}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00242},
}
Figure