In centralized deep learning, adaptive optimizers like Adam, RMSprop, and Lion are indispensable. By tracking historical gradients to scale updates per-parameter, they allow models to navigate complex, non-convex loss landscapes—especially those common in modern architectures like Transformers.
When we transition to Federated Learning (FL), where training is distributed across decentralized devices, applying these adaptive insights becomes remarkably challenging. If individual clients run adaptive optimizers locally on their highly heterogeneous (non-IID) data, their optimization trajectories rapidly diverge. Simply averaging these drifted parameters at a central server destroys the mathematical benefits of adaptivity.
Adaptive federated optimization solves this dilemma by shifting or coordinating the adaptive mechanism at the system level, combining the stability of centralized adaptive optimization with the privacy and efficiency of distributed learning.
The Core Mechanism: Pseudo-Gradients
To understand why adaptive federated optimization works, we must look at how the central server interprets client contributions. In standard Federated Averaging (FedAvg), clients perform multiple steps of local Stochastic Gradient Descent (SGD) and send their updated weights back to the server, which computes a simple average.
Adaptive federated optimization re-frames this aggregation step. Instead of viewing the received client updates merely as new coordinates, the server treats the collective change in model weights as a pseudo-gradient.
If $W_t$ is the global model at round $t$, and $\Delta W_t$ is the average parameter change returned by the participating clients after their local training iterations, the server treats $-\Delta W_t$ as a noisy gradient vector. The server can then feed this pseudo-gradient into a global, server-side adaptive optimizer.
Clients (Local Training) Server Aggregation Server-Side Optimizer
[Client 1: Local SGD] ---\ .-------------------------.
[Client 2: Local SGD] ----+---> Compute Delta W (Pseudo-Gradient) -> | Tracks First/Second | -> Global Update
[Client 3: Local SGD] ---/ | Moments (Adam, Lion...) |
'-------------------------'
As noted by Jingyi Chai et al. in FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models, this mechanism allows the server to adjust global updates based on the variance and momentum of client contributions. By maintaining moving averages of these pseudo-gradient moments at the server level, the system mitigates the challenges of non-IID data distributions across clients without requiring clients to transmit their own local optimizer states.
Why Server-Side Adaptivity Stabilizes Training
Shifting adaptivity to the server provides two primary benefits: it smooths out the optimization trajectory and filters out system-induced noise.
Inducing Subspace Smoothness
In complex tasks like Automatic Speech Recognition (ASR) or Natural Language Processing, standard federated gradient methods often suffer from severe instability. [Sheikh Shams Azam](/author/sheikh-shams-azam) et al. demonstrate in Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASR that server-side adaptive optimizers enhance convergence by increasing the smoothness of the gradient subspace across training rounds. This effectively neutralizes the high-frequency noise generated by highly divergent client updates.
Filtering Heterogeneity and Privacy Noise
Federated systems often introduce additional layers of volatility, such as client sampling jitter and Differential Privacy (DP) noise added to protect user data. Lin Chen et al. show in Differentially Private Federated Learning on Non-iid Data: Convergence Analysis and Adaptive Optimization that by scaling updates based on historical pseudo-gradients, server-side adaptive mechanisms smooth out the drift and jitter that typically hinder convergence in private federated learning.
The theoretical foundation for these "outer optimizers" is detailed by Ahmed Khaled in Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration, proving that the server-side step size acts as a critical knob for balancing optimization speed against gradient noise.
Communication Efficiency and Compression
A naive implementation of adaptive learning in a distributed setting might involve clients maintaining and transmitting their own optimizer buffers (like the first and second moments in Adam). However, this doubles or triples the communication overhead per round.
Server-side adaptivity inherently avoids this overhead because the tracking states reside entirely on the server. For example, [Tsung-Hui Chang](/author/tsung-hui-chang) et al. introduce a framework in FedLion: Faster Adaptive Federated Optimization with Fewer Communication that leverages sign-based local updates aggregated at the server, capturing the benefits of adaptive learning rates while minimizing the volume of data transmitted.
Furthermore, server-side adaptive optimization is highly compatible with lossy gradient compression and sketching techniques:
- Error Feedback:
Xiaoyun Li et al.demonstrate in Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression: Fast Convergence and Partial Participation that integrating error-compensated compression with server-level adaptive states allows systems to achieve substantial communication reductions while maintaining convergence stability. - Sketching: When updates are compressed via sketching to preserve privacy,
[Qiaobo Li](/author/qiaobo-li) et al.show in Sketched Gaussian Mechanism for Private Federated Learning that server-side adaptive methods effectively compensate for the resulting information loss, outperforming uncompressed gradient descent across various privacy levels.
Balancing Server and Client Adaptivity
While server-side adaptivity manages global convergence, can clients still benefit from local adaptivity?
[Sohom Mukherjee](/author/sohom-mukherjee) et al. point out in Locally Adaptive Federated Learning that restricting adjustments solely to the central server prevents individual clients from responding to their specific local gradient curvatures during local training loops.
To address this, the field has moved toward understanding the synergy between both levels. [Junhyung Lyle Kim](/author/junhyung-lyle-kim) et al. demonstrate in Adaptive Federated Learning with Auto-Tuned Clients that client-level step-size auto-tuning and server-level adaptive optimization are non-redundant. When combined, they provide synergistic benefits, allowing individual nodes to navigate local data geometry while the server stabilizes the collective trajectory.
Extrapolation as an Adaptive Tool
A closely related facet of adaptive optimization is the use of dynamic server-side step sizes determined by extrapolation. Instead of taking a standard step along the averaged client update vector, the server adaptively extrapolates "further" along the update direction based on the alignment of client trajectories.
[Divyansh Jhunjhunwala](/author/divyansh-jhunjhunwala) et al. show in FedExP: Speeding Up Federated Averaging via Extrapolation that implementing adaptive server-side stepsizes through extrapolation can significantly accelerate convergence without requiring extra client-side computation.
Building on this principle, [Hanmin Li](/author/hanmin-li) et al. demonstrate in The Power of Extrapolation in Federated Learning that semi-adaptive extrapolation strategies—which monitor the alignment of client updates and increase step sizes when updates are diverse—allow practitioners to optimize convergence dynamically without needing to pre-calculate complex data smoothness constants.
Key Foundational Work
Adaptive Federated Optimization (2020) — contributed 2 concept elements to this direction.
Sparsified Privacy-Masking for Communication-Efficient and Privacy-Preserving Federated Learning (2020) — contributed 2 concept elements to this direction.
Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization (2020) — contributed 2 concept elements to this direction.