Topology-Driven Synchronization Interval Optimization for Latency-Constrained Geo-Decentralized Federated Learning

Qi Chen, Wei Yu, Xinchen Lyu, Zimeng Jia, Guoshun Nan, Qimei CuiIEEE Open J. Commun. Soc 2024 · OpenReview

Abstract

Geo-decentralized federated learning (FL) can empower fully distributed model training for future large-scale 6G networks. Without the centralized parameter server, the peer-to-peer model synchronization in geo-decentralized FL would incur excessive communication overhead. This paper first proposes the synchronization interval optimization for latency-constrained geo-decentralized FL. The problem is formulated to maximize the model training accuracy within a time window under communication/computation constraints. We mathematically derive the convergence bound by jointly considering data heterogeneity, network topology and communication/computation resources. By minimizing the convergence bound, we optimize the synchronization interval based on the approximated system consistency metric. Extensive experiments on MNIST, Fashion-MNIST and CIFAR10 datasets validate the superiority of the proposed approach by achieving up to 30% higher accuracy than the state-of-the-art benchmarks.

Introduction

Traditional Federated Learning (FL) relies on a central "Parameter Server" to aggregate model updates from various clients. However, in massive 6G networks, this centralized approach creates bottlenecks and single points of failure. Geo-decentralized FL solves this by allowing edge servers to sync only with their immediate neighbors (peer-to-peer), enhancing scalability and robustness.

The challenge lies in the fact that peer-to-peer syncing is slow and bandwidth-heavy. While previous research focused on making this process more "efficient," they often ignored real-world latency constraints. In many practical scenarios, a model must be trained within a fixed time window (e.g., before new data arrives or a task expires). If too much time is spent syncing, training time is lost; if syncing is too infrequent, the models across the network diverge. This paper addresses the "Goldilocks" problem: finding the optimal synchronization interval to maximize accuracy before the clock runs out.

Methodology

The authors propose a Topology-Driven Synchronization Interval Optimization approach. Their methodology follows three main pillars:

  1. Convergence Bound Derivation: An upper bound for the loss function was mathematically derived, explicitly accounting for network topology (connectivity), data heterogeneity (local dataset differences), and available hardware resources (computation and communication speeds).
  2. System Consistency Metric: To understand model drift across the network without a central server, the authors introduced a "System Consistency" metric. They extended Dijkstra’s algorithm to estimate model differences based on the shortest paths between nodes in the network graph.
  3. Adaptive Interval Control: The problem of finding the optimal interval ($\tau$) is formulated as an integer programming problem, solved in real-time using Particle Swarm Optimization (PSO). The algorithm is adaptive, recalculating the optimal sync interval as training progresses and system parameters shift.
Figure 4: System model for geo-decentralized FL showing the synchronization timeline within a fixed time window Z.

Experimental Setup

The approach was evaluated using three standard computer vision benchmarks:

  • MNIST: Hand-written digits.
  • Fashion-MNIST (FMNIST): A complex dataset of clothing items.
  • CIFAR10: Color images across 10 classes.

To simulate real-world decentralized conditions, datasets were distributed across 10 edge servers using a Dirichlet distribution to create "Non-IID" (non-independent and identically distributed) data, mimicking geographic data variance.

Key Insights

  • Topology is Destiny: "Algebraic Connectivity" ($\alpha$) is a critical lever. In denser networks (e.g., a grid), servers can sync less frequently because information spreads faster. In sparse networks (e.g., a ring), frequent syncing is vital to prevent local models from diverging.
  • The Sync-Train Trade-off: In latency-constrained environments, the optimal interval maximizes the number of useful gradient steps taken before the deadline, rather than just saving bandwidth.
  • Dijkstra for Model Drift: Treating model divergence like a routing problem allows for estimating the global state of the network without a central coordinator.

Results

The proposed "Topology-Driven" approach was compared against standard Decentralized Stochastic Gradient Descent (DSGD) and DFedAvgM.

  • Accuracy Gains: The method achieved up to 30.65% higher accuracy than DSGD on MNIST and outperformed DFedAvgM across all datasets.
  • Robustness to Bandwidth: When bandwidth was halved (from 1MB/s to 0.5MB/s), the algorithm's advantage increased, demonstrating its ability to prioritize computation when communication is expensive.
  • Adaptability: Experimental results show that the algorithm naturally chooses larger synchronization intervals early in training and adjusts as the system reaches a more consistent state.

Conclusion and Future Work

While the results are impressive, future work includes automating the selection of the "correction factor" ($\lambda$) to make the system more "plug-and-play." Additionally, extending the framework to dynamic topologies (e.g., mobile 6G nodes like drones) and exploring asynchronous updates to handle heterogeneous latency could further improve efficiency.

Summary updated

Abstract

Geo-decentralized federated learning (FL) can empower fully distributed model training for future large-scale 6G networks. Without the centralized parameter server, the peer-to-peer model synchronization in geo-decentralized FL would incur excessive communication overhead. Some existing studies optimized synchronization interval for communication efficiency, but may not be applicable to latency-constrained geo-decentralized FL. This paper first proposes the synchronization interval optimization for latency-constrained geo-decentralized FL. The problem is formulated to maximize the model training accuracy within a time window under communication/computation constraints. We mathematically derive the convergence bound by jointly considering data heterogeneity, network topology and communication/computation resources. By minimizing the convergence bound, we optimize the synchronization interval based on the approximated system consistency metric. Extensive experiments on MNIST, Fashion-MNIST and CIFAR10 datasets validate the superiority of the proposed approach by achieving up to 30% higher accuracy than the state-of-the-art benchmarks.
BibTeX
@article{chen2024topology,
  author = {Qi Chen and Wei Yu and Xinchen Lyu and Zimeng Jia and Guoshun Nan and Qimei Cui},
  title = {{Topology-Driven Synchronization Interval Optimization for Latency-Constrained Geo-Decentralized Federated Learning}},
  journal = {IEEE Open J. Commun. Soc.},
  year = {2024},
  url = {https://openreview.net/forum?id=chw19coggI},
}
Figure