Smart Sampling: Helping from Friendly Neighbors for Decentralized Federated Learning

Lin Wang, Yang Chen, Yongxin Guo, Xiaoying TangarXiv 2024 · DOI · arXiv

Abstract

Decentralized Federated Learning (DFL) removes the central server bottleneck of traditional federated learning, allowing clients to communicate peer-to-peer. However, data heterogeneity across nodes often leads to "unfriendly" collaborations that can degrade model performance. We present AFIND+ (Adaptive FrIendly Neighbor Discory), an algorithm designed to optimize neighbor selection. AFIND+ utilizes feature-based proxies to identify compatible collaborators, adaptively tunes participation through entropy-based thresholds, and reweights model contributions during aggregation. Our evaluations show that AFIND+ consistently outperforms current state-of-the-art DFL methods in both accuracy and convergence speed.

Introduction

In Decentralized Federated Learning (DFL), the absence of a central orchestrator enhances privacy and robustness but introduces significant challenges related to data heterogeneity. When nodes have non-IID (independent and identically distributed) data, blind model averaging can be counterproductive. Collaborating with a neighbor whose data distribution is fundamentally different can actually result in worse performance than training in isolation.

To address this, a DFL system must effectively solve three problems:

  1. Identification: Detecting "friendly" neighbors without accessing raw data.
  2. Participation: Determining the optimal number of collaborators dynamically.
  3. Aggregation: Weighting neighbor contributions based on their relevance and performance.

The AFIND+ Framework

The proposed AFIND+ algorithm tackles these challenges through a three-stage pipeline: coreset-based identification, adaptive thresholding, and contribution-aware aggregation.

Overview of the AFIND+ workflow, illustrating the transition from data featurization to adaptive client sampling and model aggregation.

1. Coreset-based Identification

To minimize communication overhead while maintaining privacy, AFIND+ avoids sharing full model gradients. Instead, it uses the output of a model's featurizer layer (the layer immediately preceding the classifier) as a proxy for data distribution. By comparing these "fingerprints," nodes can identify neighbors with similar data characteristics.

2. Adaptive Thresholding

Rather than relying on a static number of neighbors, AFIND+ employs an entropy-based confidence measure. If a client receives highly diverse signals from its neighbors (high entropy), it increases its similarity threshold to become more selective. Conversely, if neighbors are consistently similar, the threshold is lowered to maximize collaborative gains.

3. Contribution-Aware Aggregation

Selected neighbors are not weighted equally. AFIND+ applies a Boltzmann distribution based on local losses to assign weights during aggregation. This ensures that models performing better on the local task have a greater influence on the updated parameters.

Experimental Evaluation

The researchers evaluated AFIND+ against standard benchmarks, including CIFAR-10, CIFAR-100, FEMNIST, and FashionMNIST. Data was partitioned using Dirichlet and Pathological distributions to simulate various non-IID scenarios.

Key Results

AFIND+ demonstrated superior performance across all tested configurations. On CIFAR-10, it achieved a 5% accuracy improvement over the next best method, while CIFAR-100 saw a 4% gain.

Comparison of AFIND+ against baseline algorithms across CIFAR-10 and CIFAR-100 datasets under different non-IID settings.

Beyond raw accuracy, AFIND+ showed:

  • Faster Convergence: Reaching target accuracy levels in significantly fewer communication rounds on the FEMNIST dataset.
  • Plug-and-Play Compatibility: The algorithm can be integrated into existing personalized FL frameworks like FedAvg-FT or Ditto for immediate performance gains.
  • Robustness: Maintaining effectiveness in partially connected networks and under Differential Privacy (DP) constraints.

Conclusion and Future Work

AFIND+ provides a robust solution for neighbor selection in decentralized networks by focusing on "friendly" collaborations. While highly effective, future research is needed to address scalability in massive networks and to improve stability in highly volatile environments where nodes frequently join or leave the network.

Summary updated

BibTeX
@misc{wang2024smart,
  author = {Lin Wang and Yang Chen and Yongxin Guo and Xiaoying Tang},
  title = {{Smart Sampling: Helping from Friendly Neighbors for Decentralized Federated Learning}},
  year = {2024},
  eprint = {2407.04460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.04460},
}
Figure