Aggregation Weighting of Federated Learning via Generalization Bound Estimation

Mingwei Xu, Xiaofeng Cao, Ivor W. Tsang, James T. KwokarXiv 2023 · DOI · OpenReview

Abstract

Federated Learning (FL) typically aggregates client models using weights proportional to their local dataset sizes. However, this "naive" approach is susceptible to performance degradation when client data is non-IID or corrupted by noise. This paper introduces a robust aggregation weighting strategy that leverages generalization bound estimation. By analyzing the distributional robustness of local models, the server calculates a "bound disagreement" for each client, assigning higher weights to those with more stable and reliable updates. Experimental results on standard benchmarks demonstrate significant improvements in accuracy and stability, particularly in high-noise and heterogeneous environments.

Introduction

In the standard Federated Learning paradigm, algorithms like FedAvg aggregate local model updates by weighting them based on the number of samples each client possesses. While computationally efficient, this approach assumes that all data samples contribute equally to the global model's performance. In real-world scenarios, datasets are often non-Independent and Identically Distributed (non-IID) and may contain significant label noise. Under these conditions, a client with a large but low-quality dataset can disproportionately bias the global model, leading to model drift and poor convergence.

This research addresses the challenge of identifying and prioritizing high-quality client updates without direct access to local data. The authors propose moving beyond sample-size weighting to a strategy based on the estimated generalization capability of local models.

Methodology: Robust Aggregation Weighting

The core of the proposed method is a dynamic weighting mechanism that replaces static sample-proportion weights with values derived from generalization bounds.

Generalization Bound Estimation

The approach utilizes the second-order origin moment of the loss function, which captures both model bias and variance. By applying principles from distributional robustness analysis, the authors estimate the "worst-case" (upper bound) and "best-case" (lower bound) performance of a local model under potential distribution shifts.

Overview of the robust aggregation process across communication rounds, showing the calculation of weights $w_j$ based on the bound disagreement $\eta_j$.

Bound Disagreement ($\eta$)

For each client $j$, the server calculates the "bound disagreement" $\eta_j$, defined as the gap between the estimated upper and lower generalization bounds. A tight disagreement indicates that the model is stable and the underlying data is relatively homogeneous. Conversely, a loose disagreement suggests high uncertainty, often stemming from noise or extreme data heterogeneity.

Estimated upper and lower bounds for classification error and JSD loss as a function of Hellinger distance $\epsilon$, illustrating the theoretical foundation for bound disagreement.

Dynamic Aggregation

In each communication round, the server assigns weights inversely proportional to the bound disagreement. This ensures that clients with more reliable, "flatter" loss landscapes have a greater influence on the global model update. This strategy is model-agnostic and can be integrated into various FL frameworks, including FedAvg, FedProx, SCAFFOLD, and FedDyn.

Experimental Evaluation

The method was evaluated using MNIST, EMNIST, CIFAR-10, and CIFAR-100 datasets. To simulate realistic federated conditions, the researchers employed:

  • Statistical Heterogeneity: Data was partitioned using a non-equilibrium Dirichlet distribution to create class imbalances among clients.
  • Label Noise: Controlled levels of label noise (20% to 40%) were introduced to test robustness against corrupted data.

Key Results

The robust weighting strategy yielded substantial improvements across multiple dimensions:

  • Accuracy: On MNIST with 10 clients, the robust version of FedAvg improved accuracy from 72.21% to 96.86%. Significant gains were also observed on the more complex CIFAR datasets.
  • Noise Robustness: The method proved highly resilient to label noise. In EMNIST scenarios with 40% noise, accuracy reached 81.72%, compared to 63.35% for traditional FedAvg.
  • Convergence Stability: The variance of test accuracy was reduced by up to 90% in some experiments, indicating smoother and more predictable convergence.
  • Algorithmic Efficiency: By improving the aggregation step, simpler algorithms like FedAvg were able to match the performance of more complex counterparts like SCAFFOLD while maintaining lower computational overhead.

Conclusion

This paper demonstrates that weighting client updates based on generalization bound estimation is a powerful alternative to traditional sample-size-based aggregation in Federated Learning. By effectively penalizing unreliable or noisy updates through the "bound disagreement" metric, the proposed method enhances global model performance and robustness in heterogeneous environments.

Summary updated

Abstract

Federated Learning (FL) typically aggregates client model parameters using a weighting approach determined by sample proportions. However, this naive weighting method may lead to unfairness and degradation in model performance due to statistical heterogeneity and the inclusion of noisy data among clients. Theoretically, distributional robustness analysis has shown that the generalization performance of a learning model with respect to any shifted distribution is bounded. This motivates us to reconsider the weighting approach in federated learning. In this paper, we replace the aforementioned weighting method with a new strategy that considers the generalization bounds of each local model. Specifically, we estimate the upper and lower bounds of the second-order origin moment of the shifted distribution for the current local model, and then use these bounds disagreements as the aggregation proportions for weightings in each communication round. Experiments demonstrate that the proposed weighting strategy significantly improves the performance of several representative FL algorithms on benchmark datasets.
BibTeX
@misc{xu2023aggregation,
  author = {Mingwei Xu and Xiaofeng Cao and Ivor W. Tsang and James T. Kwok},
  title = {{Aggregation Weighting of Federated Learning via Generalization Bound Estimation}},
  year = {2023},
  eprint = {2311.05936},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.05936},
}
Figure