Abstract
Gossip Learning (GL) provides a decentralized alternative to Federated Learning, but it suffers from slow convergence in networks characterized by homophily—the tendency for similar nodes to cluster. This paper introduces a data-driven weighting strategy that prioritizes updates from neighbors with diverse data distributions. By leveraging softmax-distribution-based weighting and temporal regularization, the proposed method significantly enhances convergence speed and model accuracy across various benchmarks and real-world datasets.
Introduction: The Challenge of Homophily in Gossip Learning
Decentralized AI often employs Gossip Learning (GL), where nodes share model updates with immediate neighbors rather than a central server. This architecture enhances privacy and robustness. However, real-world social networks are rarely uniform; they exhibit "homophily," where nodes with similar data distributions are more likely to be connected.
As shown in Figure 1, homophily creates "echo chambers." Nodes within a cluster (like A, B, C, D) repeatedly exchange similar information, while diverse knowledge from other clusters (like E, F, G, H) struggles to diffuse across sparse "bridge" edges like A-E. Traditional topological weighting schemes, such as Metropolis-Hastings, fail to account for this data imbalance, leading to poor performance.
Methodology: Data-Driven Weighting and Regularization
To overcome these bottlenecks, the authors introduce three key innovations:
1. Softmax-Distribution-Based Weighting
Instead of static topological weights, nodes dynamically evaluate the "informativeness" of their neighbors. By passing a small local validation set through a neighbor's model and comparing the resulting softmax probabilities to their own, nodes can identify neighbors with significantly different data distributions. These "bridge" neighbors are assigned higher weights, accelerating the spread of novel information.
2. Fairness through Temporal Regularization
In a decentralized setting, nodes can diverge in their learning trajectories. The authors implement a regularization term that penalizes local models for deviating too far from the previous aggregate state. This ensures that all nodes, regardless of their position in the graph, reach comparable accuracy levels simultaneously.
3. SDAH Generative Model
The researchers developed the Social Distance Attachment for Heterogeneous data (SDAH) model to synthesize realistic networks where connection probabilities are tied to data similarity, providing a robust testbed for homophily-aware algorithms.
Key Results and Discussion
The proposed "Softmax-wt" method was evaluated against standard benchmarks (MNIST, CIFAR-10) and a specialized bird classification dataset from iNaturalist.
As illustrated in Figure 5(a), the data-driven weighting scheme achieves much higher accuracy and faster convergence than the topological baseline. Figure 5(b) confirms the mechanism: the weight for the bridge edge (A-E) increases significantly over time, while weights for neighbors within the same cluster (A-B, A-C, A-D) decrease. This confirms that the algorithm successfully identifies and prioritizes the most informative connections.
In the Bird Classification task, the method reached 81% median test accuracy within 80 rounds, nearly doubling the 46% achieved by the baseline. Furthermore, the approach proved robust to reduced communication frequencies, maintaining performance even when weights were cached for multiple rounds.
Conclusion
By shifting from topological to data-driven weighting, Gossip Learning can effectively navigate the challenges of homophilous networks. This approach not only speeds up convergence but also ensures fairer outcomes across heterogeneous nodes. Future work will focus on automating hyperparameter tuning and adapting the method for dynamic, time-varying network topologies.
Summary updated