Local Learning Rate Adaptation in Heterogeneous Federated Learning

24 concepts19 papers1 proposal

In standard Federated Learning (FL) frameworks like Federated Averaging (FedAvg), decentralized clients download a global model, perform a fixed number of local Stochastic Gradient Descent (SGD) steps, and send their updates back to a central server for averaging. While elegant, this uniform approach struggles in the real world due to two main forces: systems heterogeneity (devices have varying computational speeds, power constraints, and network availability) and data heterogeneity (devices hold non-identically distributed datasets of vastly different sizes).

When clients with diverse data profiles and hardware capabilities use a single, uniform learning rate, the global model suffers from client drift and objective inconsistency. A client with a massive dataset or a faster processor will perform far more local updates, drifting deeply into its local loss landscape and pulling the global model away from the true global optimum.

To solve this, a powerful paradigm has emerged: heterogeneity-aware local learning rate adaptation. Instead of treating all clients identically or trying to fix imbalances during server-side aggregation, this approach dynamically scales the learning rate or step size at the source based on each client's unique data geometry, sample size, and computational capacity.

The Intuition of Inverse Step-Size Scaling

To understand why local learning rate adaptation works, consider the trajectory of a single client during a local training round. If Client A performs 100 local gradient steps and Client B performs 5 steps, Client A's cumulative update vector will be significantly larger. During server aggregation, Client A will disproportionately dictate the direction of the global model, causing the system to bias toward Client A's local objective.

The most direct remedy is inverse step-size scaling: adjusting the local learning rate to be inversely proportional to the volume of local work performed.

As shown by Horváth et al. in FedShuffle: Recipes for Better Use of Local Work in Federated Learning, scaling local learning rates inversely to the number of local updates or dataset size prevents clients with larger datasets from drifting excessively. The authors demonstrate that proactive scaling at the device level is mathematically superior to attempting to re-weight updates on the server side after the drift has already occurred.

This principle is reinforced across various heterogeneous environments:

By shrinking the step size when the number of iterations increases, the effective distance traveled by every client remains bounded, keeping all local models within a safe neighborhood of the global model.

Balancing System and Participation Heterogeneity

In practical edge networks, client availability fluctuates wildly. Devices may drop offline due to battery depletion, or participate asynchronously at different frequencies. Local step-size adaptation serves as a crucial buffer against these systemic imbalances.

Managing Stragglers and Asynchrony

When local training iterations are constrained by a strict time window, slower devices complete fewer steps. In FLARE: A New Federated Learning Framework with Adjustable Learning Rates over Resource-Constrained Wireless Networks, Xiao et al. propose scaling the local learning rate linearly to compensate for fewer steps. This ensures that resource-constrained edge devices still contribute a balanced, meaningful update without bottlenecking the system.

Conversely, in asynchronous or semi-asynchronous networks, fast clients can easily overwhelm the server. Jeong et al. demonstrate in Asynchronous Decentralized Learning over Unreliable Wireless Networks that tuning learning rates based on individual device update frequencies prevents the global model from biasing toward highly active participants. Similarly, Li et al. in FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning use tailored hyperparameters to dampen the learning rates of fast clients while using momentum to stabilize slow but accurate updates.

Correcting Participation Bias

Devices often exhibit correlated or non-uniform participation patterns (e.g., phones only participating when plugged into power at night). Sun et al. address this in Debiasing Federated Learning with Correlated Client Participation by adjusting local learning rates based on the inverse of a client's participation frequency. This effectively debiases the system without requiring the central server to have prior knowledge of device availability.

Furthermore, when the size of the active client cohort changes dynamically round-by-round—such as in green energy networks—Shen et al. show in On Federated Learning with Energy Harvesting Clients that scaling the learning rate in proportion to the square root of the number of active participants stabilizes the aggregate gradient variance.

Geometry-Aware and Data-Driven Adaptation

Beyond sample sizes and hardware speeds, the underlying mathematical structure of each client's local data distribution offers vital clues for optimizing step sizes.

Different data distributions create distinct loss landscapes with varying degrees of smoothness. Enforcing a uniform step size across these landscapes can cause updates on highly non-smooth functions to oscillate or diverge.

Yi et al. exploit this in Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learning by assigning individualized stepsizes to each client based on the local smoothness of their data. This allows clients to navigate their specific data geometry efficiently. On a broader scale, Richtárik et al. demonstrate in Error Feedback Reloaded: From Quadratic to Arithmetic Mean of Smoothness Constants that shifting theoretical convergence dependencies to the arithmetic mean of local smoothness constants justifies significantly larger local step sizes in heterogeneous environments, bypassing the bottlenecks caused by a few highly non-smooth clients.

To eliminate tedious hyperparameter tuning entirely, Mukherjee et al. introduce a fully uncoordinated approach in Locally Adaptive Federated Learning. By leveraging the Stochastic Polyak Stepsize (SPS), clients independently compute their learning rates based solely on the unique geometry of their local loss landscape, significantly improving generalization on non-IID data.

Leveraging Divergence and Fairness

Interestingly, data heterogeneity does not always have to be mitigated—it can sometimes be leveraged. In On the Convergence of Zeroth-Order Federated Tuning for Large Language Models, Ling et al. show that clients with higher data divergence can accelerate global convergence if they are given larger learning rates. By using a data heterogeneity index derived from convergence bounds, the system treats diverse data distributions as a catalyst for faster loss reduction.

Client Conditions         Adaptation Strategy                 Primary Benefit
-----------------------------------------------------------------------------------------
High Local Steps /        Inverse Scaling                     Prevents Client Drift &
Large Sample Size         (↓ Learning Rate)                   Objective Inconsistency

Low Computation Time /    Linear Compensation                 Allows Contribution
Fewer Local Steps         (↑ Learning Rate)                   Without Straggler Bottlenecks

High Participation        Inverse Frequency Scaling           Removes Selection Bias;
Frequency                 (↓ Learning Rate)                   Ensures Global Fairness

Highly Non-Smooth         Smoothness-Based Scaling            Stabilizes Gradients;
Loss Landscape            (↓ Learning Rate)                   Prevents Local Divergence

Finally, adaptive step sizes can emerge implicitly from optimization objectives. As explored by Zhang et al. in Proportional Fairness in Federated Learning, fairness-aware objectives naturally scale updates based on performance: clients experiencing high loss receive larger effective updates to accelerate their training, while those with low loss receive smaller updates, effectively synchronizing the progress of a highly diverse network.

Key Foundational Work

FedShuffle: Recipes for Better Use of Local Work in Federated Learning (2022) — contributed 2 concept elements to this direction.
Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning (2021) — contributed 2 concept elements to this direction.
Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sample_size_adaptive_federated_learning_1ed12fe9,
  title = {sample-size adaptive federated learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sample-size-adaptive-federated-learning/rdr_1ed12fe913424835b14df627d0ccc24d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sample_size_adaptive_federated_learning_1ed12fe9_v1,
  title = {sample-size adaptive federated learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sample-size-adaptive-federated-learning/rdr_1ed12fe913424835b14df627d0ccc24d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake