Asynchronous Decentralized Learning over Unreliable Wireless Networks

Eunjeong Jeong, Matteo Zecchin, Marios Kountouris

Abstract

Decentralized learning offers privacy and scalability benefits by removing the need for a central server. However, real-world wireless edge networks introduce challenges such as intermittent connectivity and heterogeneous device performance. This paper proposes a robust asynchronous Decentralized Stochastic Gradient Descent (DSGD) algorithm that leverages stale gradient reuse and time-varying mixing to overcome these hurdles. Experimental results demonstrate that the proposed method achieves faster convergence and higher accuracy compared to traditional synchronous approaches in unreliable network environments.

Introduction

Decentralized learning enables devices to collaboratively train a shared model while keeping data local. Despite its potential, practical deployment in wireless edge networks is hindered by two primary factors: communication failures due to fading or blockages, and computation stragglers caused by hardware variability. Traditional synchronous algorithms often suffer from the "slowest node" bottleneck, while "delay barrier" methods discard valuable data from slower participants. This work addresses these issues by introducing an asynchronous framework that maintains progress using available information, even if it is outdated.

Methodology

The proposed Asynchronous Decentralized SGD algorithm incorporates several key innovations to handle the "messy" reality of wireless networks:

Stale Gradient Reuse and Heterogeneous Learning Rates

Instead of discarding updates from slow nodes (stragglers), the algorithm allows nodes to apply gradients computed in previous iterations. To ensure convergence despite varying update frequencies across nodes, the authors mathematically tune local learning rates, ensuring the model moves toward the global optimum.

Time-Varying Mixing and AirComp

The network topology is treated as dynamic. Using a consensus strategy based on Metropolis-Hastings weighting, nodes average their models with reachable neighbors at any given time. To enhance communication efficiency, Over-the-Air Computation (AirComp) is employed, where analog signals physically overlap to perform weighted sums, reducing latency.

Illustration of time-varying network topology and communication slots using AirComp.

Experimental Evaluation

The approach was evaluated using the Fashion-MNIST dataset on a CNN model trained by 15 simulated wireless edge devices. The environment modeled Rayleigh fading channels and exponential computation time distributions.

Key Results

The theoretical analysis provides a non-asymptotic convergence guarantee of $O(T^{-1/4})$. Empirically, the asynchronous approach proves superior in constrained environments. When a strict time limit is imposed ($T_{max} = \frac{4}{5} \mathbb{E}[T_{comp}]$), Asynchronous DSGD significantly outperforms Synchronous DSGD with a delay barrier, achieving 5-10% higher accuracy within the same wall-clock time.

Accuracy vs. Wall-clock time under strict delay requirements, comparing Asynchronous DSGD with synchronous alternatives.

Conclusion

This paper demonstrates that "stale" information is often more valuable than "discarded" information in decentralized learning. By embracing asynchrony and adapting to dynamic network topologies, the proposed DSGD algorithm offers a resilient solution for wireless edge intelligence. Future work may explore sophisticated weighting for stale gradients and advanced communication compression techniques to further improve scalability.

Summary updated

Abstract

Decentralized learning enables edge users to collaboratively train models by exchanging information via device-to-device communication, yet prior works have been limited to wireless networks with fixed topologies and reliable workers. In this work, we propose an asynchronous decentralized stochastic gradient descent (DSGD) algorithm, which is robust to the inherent computation and communication failures occurring at the wireless network edge. We theoretically analyze its performance and establish a non-asymptotic convergence guarantee. Experimental results corroborate our analysis, demonstrating the benefits of asynchronicity and outdated gradient information reuse in decentralized learning over unreliable wireless networks.
BibTeX
@inproceedings{jeong2022asynchronous,
  author = {Eunjeong Jeong and Matteo Zecchin and Marios Kountouris},
  title = {{Asynchronous Decentralized Learning over Unreliable Wireless Networks}},
  booktitle = {ICC},
  year = {2022},
  url = {https://openreview.net/forum?id=qTG12r_KfU},
}
Figure