Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data

Qianqian Tong, Guannan Liang, Jinbo BiarXiv 2020 · arXiv

Introduction

Federated Learning (FL) enables decentralized devices to collaboratively train a global model while keeping data localized. Despite its promise, FL faces significant challenges, particularly when dealing with non-independent and identically distributed (non-IID) data and the tendency of adaptive methods to converge to poorly generalizing "sharp" minima. This paper introduces a family of Federated Adaptive Gradient Methods (AGMs) equipped with novel calibration schemes to address these issues.

The Challenges: Client Drift and Sharp Minima

The primary hurdles in FL are twofold. First, non-IID data across clients leads to "client drift," where local updates diverge significantly from the global objective. Second, while adaptive optimizers like Adam excel in centralized settings, they often struggle in FL by getting trapped in sharp local minima. These regions of high curvature correspond to models that fit training data well but fail to generalize to unseen data.

Visualizing the optimization landscape: (a) 3D loss surface and (b) contour plots show the paths of different optimizers. (c) Test accuracy on CIFAR-10 demonstrates the performance gap between standard and adaptive methods under varying degrees of data heterogeneity ($\alpha$).

Federated Adaptive Gradient Methods

The authors propose a framework where clients perform local stochastic gradient descent (SGD) steps (the inner loop), and the server aggregates these updates using an adaptive optimizer like Adam or AMSGrad (the outer loop). To ensure stability and generalization, they introduce a Calibration Framework:

  1. $\epsilon$-calibrated: Adjusts the $\epsilon$ parameter in the denominator of Adam to prevent learning rate explosions when gradients are small.
  2. $p$-calibrated: Replaces the square root in the second-order momentum with a $p$-th root ($p \in [1/2, 1/16]$), effectively compressing the momentum.
  3. $s$-calibrated (Softplus): Utilizes the softplus function to smooth updates. This method acts as a bridge between heavy-ball momentum (for generalization) and Adam (for speed), controlled by a smoothing parameter $\beta$.

Experimental Evaluation

The methods were tested on MNIST and CIFAR-10/100 datasets. To simulate realistic federated environments, data was partitioned using a Dirichlet distribution ($\alpha$). A smaller $\alpha$ (e.g., 0.05) represents higher non-IIDness. The researchers utilized architectures including ResNet-20 and VGGNet to validate their approach.

Key Results and Insights

The proposed calibrated AGMs demonstrate significant improvements in both convergence speed and final accuracy.

  • Superior Performance: On MNIST, $\epsilon$-FedAdam reached 98.63% accuracy, surpassing FedAvg and SCAFFOLD. In high non-IID CIFAR-10 settings, $s$-FedAdam consistently outperformed baselines.
  • Stability and Convergence: Monitoring the "Norm of Gradient" reveals that uncalibrated methods suffer from wild fluctuations, whereas calibrated AGMs converge to stable stationary points.
  • Geometric Advantage: Visualization confirms that calibrated methods successfully navigate toward "flat" minima, which are essential for robust generalization in decentralized settings.
Ablation study of calibration schemes: (a-c) $\epsilon$-FedAdam, (d-f) $p$-FedAdam, and (g-i) $s$-FedAdam. The plots show how varying hyperparameters affects training loss, test accuracy, and gradient norm stability over 2000 rounds.

Conclusion

By integrating adaptive optimization with a robust calibration framework, this work provides a powerful toolset for federated learning. The $s$-calibrated FedAdam, in particular, offers a principled way to balance the rapid convergence of adaptive methods with the superior generalization of momentum-based approaches, backed by the first theoretical convergence proofs for such methods in nonconvex, non-IID settings.

Summary updated

Abstract

Federated learning allows loads of edge computing devices to collaboratively learn a global model without data sharing. The analysis with partial device participation under non-IID and unbalanced data reflects more reality. In this work, we propose federated learning versions of adaptive gradient methods - Federated AGMs - which employ both the first-order and second-order momenta, to alleviate generalization performance deterioration caused by dissimilarity of data population among devices. To further improve the test performance, we compare several schemes of calibration for the adaptive learning rate, including the standard Adam calibrated by $ε$, $p$-Adam, and one calibrated by an activation function. Our analysis provides the first set of theoretical results that the proposed (calibrated) Federated AGMs converge to a first-order stationary point under non-IID and unbalanced data settings for nonconvex optimization. We perform extensive experiments to compare these federated learning methods with the state-of-the-art FedAvg, FedMomentum and SCAFFOLD and to assess the different calibration schemes and the advantages of AGMs over the current federated learning methods.
BibTeX
@misc{tong2020effective,
  author = {Qianqian Tong and Guannan Liang and Jinbo Bi},
  title = {{Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data}},
  year = {2020},
  eprint = {2009.06557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.06557},
}
Figure