Introduction
Federated Learning (FL) enables multiple organizations (silos) to collaboratively train a machine learning model without sharing sensitive raw data. In cross-silo FL, participants are typically a small number of reliable entities, such as hospitals or financial institutions, connected via high-speed links. Despite the reliability of these nodes, training is often hindered by the "straggler effect." The system's cycle time—the duration of a single communication round—is dictated by the slowest link in the network. Traditional static topologies, like Star or Ring, force all nodes to wait for the highest-latency connection to complete its update, leading to significant idle time and inefficient resource utilization.
Methodology: Multigraph Topology and Strategic Isolation
To address the bottleneck of network latency, this work proposes a novel multigraph topology. Unlike static simple graphs, a multigraph allows for multiple edges between nodes, which can be categorized based on their synchronization requirements. This approach moves away from rigid, simple graph structures to a more flexible, delay-aware system.
The approach follows three key phases:
- Multigraph Construction: An initial overlay graph (e.g., a Ring) is transformed into a multigraph. Edges are classified as "strongly-connected" (requiring immediate synchronization) or "weakly-connected" (allowing for asynchronous updates). The number of edges is determined by actual network latency ($d_{min}$), ensuring the topology is physically optimized for the specific hardware and geography of the silos.
- Multigraph Parsing: The complex multigraph is decomposed into a sequence of simpler "states" or subgraphs that are cycled through during training.
- Strategic Isolation: In specific states, nodes with high-latency connections are intentionally "isolated." While isolated, these nodes continue local model updates but skip the global aggregation step for that round. This prevents them from delaying the rest of the network. These nodes reconnect in subsequent rounds with updated weights, ensuring their long-term contribution to the global model.
The training process utilizes Decentralized Parallel Asynchronous SGD (DPASGD), specifically adapted to manage the dynamic connectivity states of the multigraph and maintain model consistency despite periodic isolation.
Experimental Evaluation
The proposed method was evaluated using three benchmark datasets: FEMNIST (image classification), iNaturalist (species classification), and Sentiment140 (text analysis). To simulate realistic cross-silo conditions, these datasets were mapped onto several real-world and synthetic network topologies, including Gaia, Amazon, Geant, Exodus, and Ebone.
Performance and Accuracy
The multigraph approach consistently outperformed existing state-of-the-art methods, such as MATCHA, MST, and RING, across various metrics. The most striking insight is the "less is more" paradox: strategically skipping communication for slow nodes can enhance overall system velocity without degrading the final model.
- Significant Speedup: On the FEMNIST dataset within the Exodus network, the multigraph approach reduced cycle time to 12.1ms, nearly halving the 24.7ms required by the RING topology.
- Training Efficiency: Compared to the standard STAR topology, total wall-clock training time was reduced by a factor of 18.5x to 21.9x depending on the network configuration.
- Accuracy Preservation: Crucially, the reduction in communication did not compromise model performance. On the FEMNIST/Exodus task, the multigraph achieved 71.13% accuracy, slightly exceeding the 71.05% achieved by the RING topology.
Discussion and Conclusion
The results demonstrate that the multigraph approach effectively mitigates the straggler effect. By allowing the majority of the network to proceed at a higher velocity, the total wall-clock time is drastically reduced while the final global model converges to a similar or even slightly better accuracy level than traditional synchronous methods.
However, a trade-off exists between cycle time and accuracy. If nodes spend too much time isolated, it can lead to "local bias," where silo models diverge too far from the global objective. Future research will explore dynamic adjustment of multigraph states in response to real-time network fluctuations and the impact of this topology on non-IID data distributions, which remains a common challenge in federated learning environments.
Summary updated