DISCO: Distributed Inference with Sparse Communications

Minghai Qin, Chao Sun, Jaco Hofmann, Dejan Vučinić

TL;DR

DISCO (Distributed Inference with Sparse Communications) is a novel framework that enables large deep learning models to run across multiple resource-constrained edge devices by distributing individual layers. By cleverly framing the selection of transmitted data as a neural network pruning problem, DISCO reduces communication overhead by up to 5x and accelerates inference by up to 4.7x with negligible impact on accuracy.

What Problem Does This Paper Address?

Modern Deep Neural Networks (DNNs) are massive, often requiring hundreds of megabytes of RAM—far exceeding the capacity of low-cost edge SoCs like the ESP32, which may only have a few hundred kilobytes. While we can distribute a model across multiple devices (nodes) to share the memory load, we hit a major wall: communication latency.

In standard "within-layer model parallelism," every node needs to share its intermediate results with every other node to compute the next layer. On low-bandwidth connections (like WiFi or standard Ethernet), the time spent moving data between chips can be ten to a hundred times longer than the actual computation. Previous attempts to solve this usually involved splitting the network into completely independent branches, but this often leads to a significant drop in accuracy because the branches can't "talk" to each other to refine features.

Methods

The authors introduce DISCO, which treats the problem of "what data should we send?" as an optimization task.

The Pruning Equivalence

The key innovation is the realization that selecting which features to transmit between nodes is mathematically equivalent to structured weight pruning. If Node A decides it doesn't need a specific feature from Node B, the weights in Node A’s convolutional kernels associated with that feature can simply be set to zero.

The DISCO Workflow:

  1. Dense Training: Start by training a standard, fully connected (dense) model where all nodes communicate everything.
  2. Feature Selection via Pruning: The 4D convolution tensors are projected into 2D mask matrices. The researchers then apply iterative magnitude pruning (using the $L_1$ norm) to identify and "prune" the least important communication paths (sub-rows in the weight matrix).
  3. Fine-tuning: The remaining weights are fine-tuned to recover any accuracy lost during the sparsification process.
  4. Pipelining: To further optimize, DISCO pipelines computation and communication. While a node is waiting for sparse data from a neighbor, it starts processing the local features it already has, "hiding" the communication latency.

Datasets

The authors validated DISCO across a wide range of computer vision tasks and architectures:

  • Image Classification: ImageNet (using ResNet-50, DeiT-Small, and EfficientNet-B0).
  • Object Detection: COCO2017 (using SSD300).
  • Semantic Segmentation: PASCAL VOC2012 (using DeepLabV3+).
  • Image Super-Resolution: DIV2K (using ESRGAN).

Key Insights and Ideas

  • Sparsity is a Spectrum: Unlike "independent branch" methods that send 0% of data or "dense" methods that send 100%, DISCO finds that sending just a tiny fraction (e.g., 1%) of data can bridge the accuracy gap between independent and fully connected models.
  • The Equilibrium Point: There is a theoretical "equilibrium sparsity" for every layer where the time taken for communication exactly matches the time for computation. DISCO aims for this sweet spot to maximize hardware utilization.
  • Layer-Wise Sensitivity: Earlier layers in a network often tolerate much higher communication sparsity than later layers. DISCO's non-uniform approach allows it to be aggressive where it counts and dense where it's necessary for accuracy.
  • Hardware Agnostic: The framework was tested on everything from high-end NVIDIA A100s with NV-Link to ARM Cortex-M4 processors with 1Mbps wireless links, showing consistent speedups across the board.

Key Results

The quantitative improvements are substantial across all tested tasks:

  • ResNet-50: Achieved a 4.7x speedup on two nodes with 5x less data communication and 50% less memory required per node, maintaining comparable accuracy to the original model.
  • Object Detection (SSD): Reduced inference latency by 9.2x (from 3.4s to 0.37s) while maintaining a 26% mAP.
  • Semantic Segmentation (DeepLabV3+): Achieved a 9.9x latency reduction (from 12.3s to 1.24s).
  • Accuracy Gains: Compared to prior "split-then-aggregate" methods, DISCO increased accuracy by 1.6% on ImageNet and 6.6 mIoU on semantic segmentation at similar latency levels.

Limitations and Future Work

While the paper demonstrates the effectiveness of DISCO on up to 8 nodes, the majority of the detailed analysis focuses on 2-node configurations. The authors also assume a static network environment where communication bandwidth is known and constant during the design phase.

Future work could involve:

  • Dynamic Sparsity: Adapting the communication patterns in real-time based on fluctuating network conditions.
  • Heterogeneous Nodes: Optimizing for clusters where some devices are significantly more powerful than others.
  • Beyond Vision: Applying the DISCO framework to Large Language Models (LLMs), where memory constraints are even more punishing.

Summary updated

Abstract

Deep neural networks (DNNs) have great potential to solve many real-world problems, but they usually require an extensive amount of computation and memory. It is of great difficulty to deploy a large DNN model to a single resource-limited device with small memory capacity. Distributed computing is a common approach to reduce single-node memory consumption and to accelerate the inference of DNN models. In this paper, we explore the "within-layer model parallelism", which distributes the inference of each layer into multiple nodes. In this way, the memory requirement can be distributed to many nodes, making it possible to use several edge devices to infer a large DNN model. Due to the dependency within each layer, data communications between nodes during this parallel inference can be a bottleneck when the communication bandwidth is limited. We propose a framework to train DNN models for Distributed Inference with Sparse Communications (DISCO). We convert the problem of selecting which subset of data to transmit between nodes into a model optimization problem, and derive models with both computation and communication reduction when each layer is inferred on multiple nodes. We show the benefit of the DISCO framework on a variety of CV tasks such as image classification, object detection, semantic segmentation, and image super resolution. The corresponding models include important DNN building blocks such as convolutions and transformers. For example, each layer of a ResNet-50 model can be distributively inferred across two nodes with five times less data communications, almost half overall computations and half memory requirement for a single node, and achieve comparable accuracy to the original ResNet-50 model. This also results in 4.7 times overall inference speedup.
BibTeX
@inproceedings{qin2024disco,
  author = {Minghai Qin and Chao Sun and Jaco Hofmann and Dejan Vučinić},
  title = {{DISCO: Distributed Inference with Sparse Communications}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00242},
}
Figure