Abstract
Federated Learning (FL) typically aggregates client models using weights proportional to their local dataset sizes. However, this "naive" approach is susceptible to performance degradation when client data is non-IID or corrupted by noise. This paper introduces a robust aggregation weighting strategy that leverages generalization bound estimation. By analyzing the distributional robustness of local models, the server calculates a "bound disagreement" for each client, assigning higher weights to those with more stable and reliable updates. Experimental results on standard benchmarks demonstrate significant improvements in accuracy and stability, particularly in high-noise and heterogeneous environments.
Introduction
In the standard Federated Learning paradigm, algorithms like FedAvg aggregate local model updates by weighting them based on the number of samples each client possesses. While computationally efficient, this approach assumes that all data samples contribute equally to the global model's performance. In real-world scenarios, datasets are often non-Independent and Identically Distributed (non-IID) and may contain significant label noise. Under these conditions, a client with a large but low-quality dataset can disproportionately bias the global model, leading to model drift and poor convergence.
This research addresses the challenge of identifying and prioritizing high-quality client updates without direct access to local data. The authors propose moving beyond sample-size weighting to a strategy based on the estimated generalization capability of local models.
Methodology: Robust Aggregation Weighting
The core of the proposed method is a dynamic weighting mechanism that replaces static sample-proportion weights with values derived from generalization bounds.
Generalization Bound Estimation
The approach utilizes the second-order origin moment of the loss function, which captures both model bias and variance. By applying principles from distributional robustness analysis, the authors estimate the "worst-case" (upper bound) and "best-case" (lower bound) performance of a local model under potential distribution shifts.
Bound Disagreement ($\eta$)
For each client $j$, the server calculates the "bound disagreement" $\eta_j$, defined as the gap between the estimated upper and lower generalization bounds. A tight disagreement indicates that the model is stable and the underlying data is relatively homogeneous. Conversely, a loose disagreement suggests high uncertainty, often stemming from noise or extreme data heterogeneity.
Dynamic Aggregation
In each communication round, the server assigns weights inversely proportional to the bound disagreement. This ensures that clients with more reliable, "flatter" loss landscapes have a greater influence on the global model update. This strategy is model-agnostic and can be integrated into various FL frameworks, including FedAvg, FedProx, SCAFFOLD, and FedDyn.
Experimental Evaluation
The method was evaluated using MNIST, EMNIST, CIFAR-10, and CIFAR-100 datasets. To simulate realistic federated conditions, the researchers employed:
- Statistical Heterogeneity: Data was partitioned using a non-equilibrium Dirichlet distribution to create class imbalances among clients.
- Label Noise: Controlled levels of label noise (20% to 40%) were introduced to test robustness against corrupted data.
Key Results
The robust weighting strategy yielded substantial improvements across multiple dimensions:
- Accuracy: On MNIST with 10 clients, the robust version of FedAvg improved accuracy from 72.21% to 96.86%. Significant gains were also observed on the more complex CIFAR datasets.
- Noise Robustness: The method proved highly resilient to label noise. In EMNIST scenarios with 40% noise, accuracy reached 81.72%, compared to 63.35% for traditional FedAvg.
- Convergence Stability: The variance of test accuracy was reduced by up to 90% in some experiments, indicating smoother and more predictable convergence.
- Algorithmic Efficiency: By improving the aggregation step, simpler algorithms like FedAvg were able to match the performance of more complex counterparts like SCAFFOLD while maintaining lower computational overhead.
Conclusion
This paper demonstrates that weighting client updates based on generalization bound estimation is a powerful alternative to traditional sample-size-based aggregation in Federated Learning. By effectively penalizing unreliable or noisy updates through the "bound disagreement" metric, the proposed method enhances global model performance and robustness in heterogeneous environments.
Summary updated