Standard distributed and federated learning algorithms typically operate in lockstep. In a classic framework like Federated Averaging (FedAvg), a central server distributes a global model to a fleet of edge devices. Each device is expected to perform an identical amount of local work—such as exactly $E$ local epochs or $K$ optimization steps—before sending its updated parameters back for aggregation.
While mathematically clean, this uniform workload requirement breaks down in real-world deployments. Edge networks are fundamentally heterogeneous, composed of devices with vastly different computational power, battery levels, and network bandwidths. Forcing a uniform workload creates a destructive dilemma:
- The Straggler Bottleneck: If the system waits for every node to finish its assigned steps synchronously, high-performance devices sit idle at synchronization barriers, wasting valuable compute capacity and slowing down overall training time (Kumar et al.).
- Data Bias: If the system drops slow devices ("stragglers") to maintain a fast cadence, it discards the unique local data residing on those devices, introducing severe statistical bias into the global model (Sahu et al.).
To resolve this, a powerful research direction has emerged: heterogeneous local updates. Instead of forcing hardware to conform to a rigid algorithmic cadence, the algorithm is decoupled to adapt to the available hardware. By allowing devices to perform variable amounts or types of local computation, distributed systems can eliminate straggler bottlenecks while fully preserving data diversity.
Decoupling the Local Workload
The core mechanism of this paradigm is simple: let every device do what it can within a given window. Instead of prescribing a fixed number of local iterations, frameworks allow the number of local gradient steps to vary dynamically across nodes based on their real-time resource constraints (Smith et al.; Hu et al.; Xie et al.).
Traditional Synchronous FL (Rigid Steps):
Device A (Fast): |--Step 1--|--Step 2--|--------IDLE WAITING--------| AGGREGATE
Device B (Slow): |--------Step 1--------|--------Step 2--------| AGGREGATE
Heterogeneous Local Updates (Time-Bounded):
Device A (Fast): |--Step 1--|--Step 2--|--Step 3--|--Step 4--| AGGREGATE
Device B (Slow): |--------Step 1--------| AGGREGATE
In a time-bounded synchronous setup, a global communication deadline is established. As Li et al. demonstrate, clients maximize their local training volume to meet this deadline rather than aiming for a target step count. High-performance devices utilize their compute capacity to run many local iterations, generating highly refined local updates. Meanwhile, slower devices contribute rougher, lower-precision approximations computed from fewer steps, ensuring their local data is still represented in the global model (Smith et al.).
Why Mixing Precision Works
It is intuitive that letting faster nodes do more work speeds up clock time, but why does this hodgepodge of high- and low-precision updates actually converge to a valid global model?
The intuition lies in how local computation affects model variance and gradient signal.
When a fast node performs additional local updates instead of idling, it extracts more information from its local dataset during that round. As Kumar et al. point out, this extra computation directly reduces model variance across the cluster. Instead of treating hardware heterogeneity as a system flaw to be mitigated, this approach turns it into an algorithmic advantage: the system harvests extra computation from capable nodes to accelerate the entire network's progress.
Mathematically, the challenge is ensuring that nodes performing more steps do not over-index on their local data distributions, which causes "client drift" in non-IID settings. Recent theoretical insights have proven that convergence guarantees can be preserved under these conditions. For example, Yang et al. proved that frameworks can achieve a "linear speedup" for non-convex optimization even when workers perform highly variable amounts of local work. To further stabilize this process, Wang et al. combined variable local updates with quantized transmissions and variance reduction, demonstrating that the system can aggressively cut communication overhead while maintaining fast, stable convergence.
Beyond Step Counts: Inexactness and Hybrid Optimizers
Heterogeneity-aware optimization extends beyond simply adjusting the number of gradient steps; it can also alter the mathematical nature of the update itself.
Inexact Local Solutions
Rather than requiring clients to solve a local optimization subproblem to exact precision, frameworks can treat local training as an inexact optimization process. The authors of FedADMM leverage primal-dual optimization to allow varying degrees of local inexactness. Devices compute updates to whatever level of precision their hardware permits per round, and the global dual variables naturally compensate for the discrepancies, preventing client drift without forcing slow nodes to bottleneck the system.
Hybrid Optimization Methods
In extreme cases, nodes can run entirely different optimization algorithms. Niu introduced a distributed hybrid primal-dual framework (DISH) where high-performance nodes execute computationally intensive second-order Newton-type steps, while lower-power devices stick to lightweight, first-order gradient steps. This allows the system to exploit advanced optimization capabilities where they exist without shutting out resource-constrained nodes.
Coordinate-Level Redundancy
Heterogeneity can also be managed by breaking up the model architecture itself. Wang et al. proposed varying the redundancy levels across individual coordinates or blocks of coordinates. This allows the central system to harvest partial gradient updates from slower nodes rather than treating their contribution as an all-or-nothing proposition.
Extension to Reinforcement Learning and Decentralized Networks
The principles of heterogeneous local updates have expanded well beyond standard supervised learning into more complex environments.
Federated Reinforcement Learning
In federated reinforcement learning (FRL), agents learn a shared policy while interacting with distinct environments. Zhu et al. and Zhu et al. extended flexible local updates to Temporal Difference (TD) learning and Actor-Critic frameworks. By explicitly permitting agents to perform a variable number of local TD updates between communication rounds, they achieved asymptotically vanishing convergence error without synchronizing on the slowest agent.
Decentralized Sporadic Networks
In fully decentralized, peer-to-peer networks where no central server coordinates deadlines, hardware and bandwidth variations are even more pronounced. Zehtabi et al. developed a framework for decentralized sporadic federated learning ($\texttt{DSpodFL}$), modeling both local computations and peer-to-peer communications as intermittent, probabilistic events.
Their findings highlight a key rule of thumb for modern distributed optimization: the benefit of flexible local updates scales with the messiness of the system. In highly skewed environments, embracing sporadic and heterogeneous participation yields a 10% to 40% improvement in final model accuracy over rigid, synchronous baselines for a fixed physical time budget.
Key Foundational Work
Federated Multi-Task Learning (2017) — contributed 2 concept elements to this direction.
Fugue: Slow-Worker-Agnostic Distributed Learning for Big Models on Big Data (2014) — contributed 1 concept elements to this direction.