Introduction
Federated Learning (FL) enables decentralized devices to collaboratively train a global model while keeping data localized. Despite its promise, FL faces significant challenges, particularly when dealing with non-independent and identically distributed (non-IID) data and the tendency of adaptive methods to converge to poorly generalizing "sharp" minima. This paper introduces a family of Federated Adaptive Gradient Methods (AGMs) equipped with novel calibration schemes to address these issues.
The Challenges: Client Drift and Sharp Minima
The primary hurdles in FL are twofold. First, non-IID data across clients leads to "client drift," where local updates diverge significantly from the global objective. Second, while adaptive optimizers like Adam excel in centralized settings, they often struggle in FL by getting trapped in sharp local minima. These regions of high curvature correspond to models that fit training data well but fail to generalize to unseen data.
Federated Adaptive Gradient Methods
The authors propose a framework where clients perform local stochastic gradient descent (SGD) steps (the inner loop), and the server aggregates these updates using an adaptive optimizer like Adam or AMSGrad (the outer loop). To ensure stability and generalization, they introduce a Calibration Framework:
- $\epsilon$-calibrated: Adjusts the $\epsilon$ parameter in the denominator of Adam to prevent learning rate explosions when gradients are small.
- $p$-calibrated: Replaces the square root in the second-order momentum with a $p$-th root ($p \in [1/2, 1/16]$), effectively compressing the momentum.
- $s$-calibrated (Softplus): Utilizes the
softplusfunction to smooth updates. This method acts as a bridge between heavy-ball momentum (for generalization) and Adam (for speed), controlled by a smoothing parameter $\beta$.
Experimental Evaluation
The methods were tested on MNIST and CIFAR-10/100 datasets. To simulate realistic federated environments, data was partitioned using a Dirichlet distribution ($\alpha$). A smaller $\alpha$ (e.g., 0.05) represents higher non-IIDness. The researchers utilized architectures including ResNet-20 and VGGNet to validate their approach.
Key Results and Insights
The proposed calibrated AGMs demonstrate significant improvements in both convergence speed and final accuracy.
- Superior Performance: On MNIST, $\epsilon$-FedAdam reached 98.63% accuracy, surpassing FedAvg and SCAFFOLD. In high non-IID CIFAR-10 settings, $s$-FedAdam consistently outperformed baselines.
- Stability and Convergence: Monitoring the "Norm of Gradient" reveals that uncalibrated methods suffer from wild fluctuations, whereas calibrated AGMs converge to stable stationary points.
- Geometric Advantage: Visualization confirms that calibrated methods successfully navigate toward "flat" minima, which are essential for robust generalization in decentralized settings.
Conclusion
By integrating adaptive optimization with a robust calibration framework, this work provides a powerful toolset for federated learning. The $s$-calibrated FedAdam, in particular, offers a principled way to balance the rapid convergence of adaptive methods with the superior generalization of momentum-based approaches, backed by the first theoretical convergence proofs for such methods in nonconvex, non-IID settings.
Summary updated