Abstract
Distributed inference of Deep Neural Networks (DNNs) on low-power edge devices is often limited by a "communication wall," where the time spent transferring data between nodes far exceeds computation time. DISCO (Distributed Inference with Sparse Communications) addresses this bottleneck by employing within-layer model parallelism combined with sparse feature map exchange. By framing feature selection as a structured weight pruning problem, DISCO reduces communication overhead by up to 90% while maintaining high accuracy across various computer vision tasks, including classification, detection, and segmentation.
Introduction
Modern DNNs are increasingly massive, often requiring memory resources that far exceed the capabilities of individual edge devices like microcontrollers or smart cameras. While distributing the workload across multiple devices is a natural solution, standard model parallelism requires every node to receive nearly all feature data from the previous layer to compute its portion of the next. On constrained edge networks (e.g., Wi-Fi or Bluetooth), this results in a communication bottleneck where latency is dominated by data transmission rather than processing.
Methodology: Sparse Within-Layer Parallelism
DISCO introduces a novel approach to within-layer model parallelism that prioritizes communication efficiency. Instead of the traditional "dense" exchange where all nodes share all calculated features, DISCO transmits only a carefully selected subset of feature maps.
The Pruning Equivalence
The core technical innovation of DISCO is the realization that selecting which features to transmit is mathematically equivalent to structured weight pruning. If a node does not receive a specific feature from a peer, the weights associated with that feature in the node's local calculation are effectively zeroed. The DISCO pipeline follows three main steps:
- Identify Importance: Use the $L_1$ norm to identify the least important weights corresponding to cross-node communication.
- Prune and Fine-tune: Prune these weights to enforce sparse communication and fine-tune the remaining parameters to recover model accuracy.
- Latency Hiding: Leverage the reduced data volume to pipeline communication, sending required data for subsequent layers while current computations are still in progress.
Experimental Results
The researchers evaluated DISCO using diverse datasets, including ImageNet (classification), COCO2017 (object detection), PASCAL VOC2012 (segmentation), and DIV2K (super-resolution).
Key Findings
- Communication Efficiency: DISCO achieved a 90% reduction in communication for SSD object detection and a 95% reduction for DeepLabV3+ semantic segmentation with negligible performance loss.
- Latency Advantages: Unlike "Pipeline Parallelism," which processes layers sequentially across nodes and increases total latency, DISCO's within-layer approach parallelizes the work of every layer. As shown in the performance analysis, DISCO maintains low inference latency even as the number of nodes (N) increases, while simultaneously improving accuracy.
- The "Goldilocks" Zone: The study found that adding even a tiny amount of communication (e.g., 1%) back into a completely split model can significantly boost accuracy, proving that sparse communication is superior to both fully dense and fully independent architectures.
Conclusion
DISCO effectively breaks the communication bottleneck for distributed AI at the edge. By treating communication as a pruning task, it enables large-scale DNN inference on resource-constrained hardware without the prohibitive latency costs of traditional distribution methods. Future work may explore more advanced pruning techniques or heterogeneous network environments to further optimize these distributed systems.
Summary updated