The effect of network topologies on fully decentralized learning: a preliminary investigation

Luigi Palmieri, Lorenzo Valerio, Chiara Boldrini, Andrea Passarella

Introduction

Standard Federated Learning (FL) typically operates on a "star" topology, where a central server coordinates model updates from multiple clients. While effective, this architecture introduces a single point of failure and potential communication bottlenecks. Decentralized Federated Learning (DFL) offers an alternative by removing the central coordinator, allowing nodes to collaborate exclusively with their immediate neighbors in a peer-to-peer graph.

However, the transition to a fully decentralized system introduces a new variable: network topology. This paper investigates how the structural "shape" of a network—whether it is a random mesh, a social-like network with hubs, or a set of isolated communities—dictates the efficiency of knowledge propagation, particularly when data is non-identically and independently distributed (non-IID).

Methodology

The researchers employed the Decentralized Averaging (DecAvg) algorithm, a peer-to-peer adaptation of the standard FedAvg. In this framework, each node follows a three-step cycle in every communication round:

  1. Aggregation: The node averages its local model weights with the weights received from its direct neighbors.
  2. Local Training: The node performs training on its private dataset.
  3. Dissemination: The updated weights are shared back with neighbors.

To evaluate the impact of network structure, the study simulated three classic graph models involving 100 nodes:

  • Erdos-Renyi (ER): A random graph where connections between any two nodes exist with a fixed probability.
  • Barabasi-Albert (BA): A scale-free network characterized by a few highly connected "hubs," mimicking the structure of social networks.
  • Stochastic Block Model (SBM): A modular network where nodes form tight-knit communities with sparse links between different groups.

Experimental Setup and Data Distribution

The study utilized the MNIST dataset of handwritten digits. To simulate the challenges of decentralized learning, the researchers partitioned the data non-uniformly across two groups:

  • G1 vs. G2: Most nodes (90%) received only digits 0-4 (G1), while a small minority (10%) received the full set or the remaining digits 5-9 (G2).
  • Placement Strategies: The researchers tested "hub-focused" scenarios (where the most connected nodes held the rare G2 data) and "edge-focused" scenarios (where the least connected nodes held the rare data).
  • Community Constraints: In SBM models, each community was restricted to seeing only two specific digits, testing the ability of knowledge to cross modular boundaries.

Key Insights: The Mechanics of Knowledge Spread

The "Dragging" Effect

One of the paper's primary findings is the "dragging" effect caused by high-degree nodes. In DFL, hubs act as gravity wells. If a hub possesses poor or incomplete data, it effectively "drags" its neighbors toward its own mediocre performance. In edge-focused scenarios, these hubs can actually block the spread of knowledge from expert leaf nodes to the rest of the network.

Knowledge Dilution

Counter-intuitively, higher connectivity is not always beneficial. In an averaging-based system, if an "expert" node (one with rare data) is connected to too many "ignorant" nodes, its unique knowledge is diluted during the averaging process. This dilution can occur before the expert node has sufficient time to influence the global model, suggesting that the rate of information spread must be balanced against the stability of local learning.

The Community Trap

In SBM topologies, knowledge tends to be "sticky." While nodes within a single community quickly reach a consensus, the lack of robust bridges between communities prevents the system from learning the full dataset. The researchers observed that while a network might be connected enough to pass simple messages, merging complex neural network weights requires much more specific structural conditions.

Results and Performance

  • Hubs as Accelerators: When hubs possess the full dataset (hub-focused BA or ER), the network converges to high accuracy significantly faster and more uniformly than other configurations.
  • Accuracy Ceilings: In community-based SBM tests, nodes frequently hit an accuracy ceiling of approximately 25%. They successfully identified the two classes present in their local community but failed to learn the other six classes due to the structural isolation of the knowledge.
  • Variance in the Edges: In edge-focused ER networks, the standard deviation of accuracy across nodes remained high. While some nodes managed to "escape" the dragging effect of hubs to reach higher accuracy, the network as a whole struggled to reach a stable consensus.

Conclusion and Future Directions

This preliminary investigation highlights that in decentralized learning, the distribution of data relative to the network's topology is as important as the learning algorithm itself. While hubs can accelerate learning when they are well-informed, they can also act as bottlenecks that dilute rare knowledge.

Future work will expand this research to larger scales (thousands of nodes) and more complex datasets like CIFAR-10. Additionally, the authors plan to investigate how the semantic similarity between classes assigned to different communities affects the difficulty of cross-community learning and how dynamic, non-static topologies might mitigate the "dragging" effect.

Summary updated

BibTeX
@inproceedings{palmieri2023effect,
  author = {Luigi Palmieri and Lorenzo Valerio and Chiara Boldrini and Andrea Passarella},
  title = {{The effect of network topologies on fully decentralized learning: a preliminary investigation}},
  booktitle = {NetAISys@MobiSys},
  year = {2023},
  doi = {10.1145/3597062.3597280},
}
Figure