Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge

May Malka, Erez Farhan, Hai Morgenstern, Nir ShlezingerIEEE Trans. Wireless Communications 2025 · DOI

TL;DR

This paper introduces "edge ensembles," a novel approach for decentralized, low-latency collaborative inference on edge devices. By forming ensembles of compact DNNs across multiple devices, they achieve improved accuracy and robustness, even under limited connectivity, outperforming individual local inference and sometimes even larger centralized models.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have revolutionized many fields, but their computational demands traditionally confine them to powerful cloud servers. This creates a bottleneck for edge devices (like smartphones, sensors, or vehicles) that collect data in real-time and could benefit from on-device inference. While edge devices are becoming more capable, they still have limited computational resources. Existing solutions like model compression (pruning, quantization) reduce accuracy, and computation offloading or collaborative intelligence often rely on constant connectivity to a central server, introducing latency and single points of failure. This paper addresses the challenge of enabling accurate and low-latency DNN inference on edge devices by leveraging collaboration among multiple devices, even when connectivity is intermittent.

Methods

The core of the proposed solution is "edge ensembles," inspired by deep ensembles. Instead of a single large DNN, multiple edge devices each run a compact DNN. During inference, these devices collaborate to form an ensemble, aggregating their individual predictions.

Key methodological contributions include:

  • Shared Encoder, User-Specific Decoders: To facilitate collaboration while maintaining diversity (crucial for ensembles), the DNN architecture is split. A shared encoder and a trainable quantizer are common across devices, processing the input and producing quantized features. Each device then has its own unique decoder to interpret these features.
  • Quantized Feature Exchange: To mitigate communication overhead and latency on rate-limited channels, only quantized features are exchanged between devices, rather than raw data or intermediate layer outputs. A VQ-VAE (Vector Quantized Variational Autoencoder) architecture is used for efficient, trainable quantization.
  • Two Inference Protocols:
    • Algorithm 1: All participating users process the quantized features and aggregate their results. This is suitable for scenarios where all devices have similar capabilities.
  • Algorithm 2: The inferring user also processes the raw input locally using a separate decoder, while neighbors process quantized features. This algorithm uses a weighted aggregation scheme to balance the contributions of local (unquantized) and collaborative (quantized) inferences, offering better accuracy in limited connectivity.
  • Latency Analysis: The paper provides a theoretical analysis of the inference latency, considering communication and computation delays, and shows that the parallel nature of ensembles leads to only a minor additional delay compared to local inference.

Datasets

The experiments were conducted using the following image classification datasets:

  • CIFAR-10
  • CIFAR-100
  • ImageWoof

Key Insights and Ideas

  • "Wisdom of Crowds" for Edge AI: The paper effectively applies the "wisdom of crowds" concept to edge AI, showing that aggregating predictions from multiple, diverse, and potentially less powerful edge devices can yield better results than any single device alone.
  • Quantization as a Communication Enabler: Instead of being solely a compression technique, learned quantization (VQ-VAE) is leveraged as a critical component for efficient and low-latency communication in decentralized settings.
  • Diversity is Key: The success of edge ensembles hinges on the diversity of the individual models. This diversity is achieved through techniques like random initialization or data partitioning during training.
  • Trade-off Management: Algorithm 2 explicitly addresses the trade-off between accuracy and communication cost by intelligently weighting local (unquantized) and collaborative (quantized) inferences, proving beneficial in low-connectivity scenarios.
  • Outperforming Centralized Models: Surprisingly, in some scenarios, a collaborative ensemble of compact edge models can achieve higher accuracy than a single, larger centralized model, even if the centralized model has more total parameters.

Key Results

  • Accuracy Improvement: Collaboration via edge ensembles significantly improves accuracy compared to individual local inference. For example, with 16 users and 4-bit quantization, accuracy increased by over 6% compared to a single model.
  • Low Latency: The proposed approach incurs only a minor additional latency compared to local inference, thanks to parallel processing and quantized feature exchange.
  • Performance with Limited Connectivity: Even with low connectivity probabilities (e.g., p=0.2), collaboration substantially boosts accuracy, demonstrating robustness.
  • Parameter Efficiency: In some cases, three edge devices with compact models (totaling ~2.10^6 parameters) outperformed a centralized MobilenetV2_1.0 model (~2.23.10^6 parameters).
  • Algorithm 2 Advantage: Algorithm 2, which uses a weighted aggregation of local and collaborative inferences, showed a 3% accuracy improvement over a single unquantized model with 16 users on Imagewoof, outperforming Algorithm 1 in similar scenarios.
  • High-Dimensional Data Benefits: The benefits of edge ensembles become more pronounced for high-dimensional data (like Imagewoof images) where compression is more critical.

Limitations and Future Work

  • Privacy Enhancement: While quantized features offer some privacy benefits, the paper acknowledges that further work is needed to explore privacy enhancement capabilities and their combination with secure aggregation methods.
  • Security Considerations: The potential impact of malicious users on prediction accuracy is mentioned as a challenge that could be addressed by Byzantine robust aggregation methods.
  • Multi-hop Communication: Extending the algorithms to support multi-hop communications for larger ensembles is suggested, though this would likely increase latency.
  • Training Complexity: Designing diverse models for the ensemble requires careful consideration during the training phase, which is an area for further exploration.
  • Hyperparameter Tuning: The paper mentions setting aggregation coefficients and other hyperparameters, implying that optimal tuning could be a factor in achieving peak performance.

Summary updated

BibTeX
@article{malka2025decentralized,
  author = {May Malka and Erez Farhan and Hai Morgenstern and Nir Shlezinger},
  title = {{Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2025},
  doi = {10.1109/twc.2024.3497167},
}
Figure