In standard federated learning, multiple decentralized clients collaborate to train a shared model without exposing their private data. When applied to reinforcement learning (RL), this paradigm allows a fleet of autonomous agents—such as drones, robots, or wind turbines—to pool their experiences and learn optimal behaviors far more quickly than any single agent could alone.
However, standard distributed RL frameworks rely on a fragile assumption: environmental homogeneity. They assume that every agent interacts with the exact same underlying Markov Decision Process (MDP), sharing identical transition dynamics and reward structures as noted by Agarwal et al. and Zheng et al..
In the real world, this assumption immediately breaks down. A fleet of delivery drones faces varying wind conditions; mechanical robots suffer from unique hardware wear and sensor miscalibrations; and regional microgrids operate under distinct physical constraints and resource distributions. This phenomenon is known as environmental heterogeneity. When agents face non-identical transition dynamics, forcing them to converge on a single global policy causes the training process to degrade or fail entirely.
The Mechanics of Failure: Client Drift and the Error Floor
To understand why environmental heterogeneity breaks standard federated reinforcement learning, we must look at how local updates interact at the central server.
In a homogeneous setting, local policy gradients or Q-value updates all point toward a shared global optimum. But when environments differ, an action that is highly advantageous for Agent A might be catastrophic for Agent B. For instance, if Agent A is a drone flying through a high-wind corridor, its local updates will aggressively adjust to compensate for aerodynamic drift. If Agent B is operating in a calm indoor warehouse, its updates will optimize for precision maneuvers.
When these updates are aggregated by a central server, they pull the global model in conflicting directions—a phenomenon known as client drift. As identified by Xie et al. and Wang et al., client drift introduces severe training instability and can cause the global policy to oscillate or diverge.
Agent A Update (High Wind) ───>
└───> Aggregated Global Policy (Suboptimal for both)
Agent B Update (Calm Indoor) ──>
Even when federated algorithms do manage to converge under heterogeneous conditions, they hit a fundamental theoretical bottleneck: a permanent suboptimality gap or error floor.
As proven by Jin et al. and Zhang et al., a single global policy trained across diverse environments can only converge to a near-optimal region of a theoretical "average" or "mixture" MDP. The radius of this remaining error ball is directly proportional to the degree of environmental diversity, which researchers quantify as the distance between the local transition probability distributions. If the environments are too diverse, the shared global policy becomes useless for everyone. Furthermore, Jin et al. demonstrated that this heterogeneity makes the optimal policy highly sensitive to initial state distributions, adding another layer of instability that is absent in standard, single-environment RL.
Algorithmic Solutions: How to Collaborate Safely
To overcome the error floor without sacrificing the benefits of collaborative learning, recent research has shifted toward algorithms that explicitly accommodate or penalize environmental divergence.
1. Alignment via Divergence Penalties
One direct way to counter client drift is to anchor local updates to the global consensus. In the FedKL framework proposed by Xie et al., the local optimization objective is modified to include a Kullback-Leibler (KL) divergence penalty between the agent's local policy and the global policy. This penalty acts as a soft constraint: it allows agents to adapt to their local physics while preventing them from drifting so far that they overwrite or corrupt the collective knowledge base during aggregation.
2. Decoupling Representation from Policy
Instead of forcing agents to share a single end-to-end policy network, why not share only the parts of the network that understand the structure of the world?
The PFedRL-REP framework introduced by Xiong et al. splits the model into a shared feature representation layers and personalized local policy heads. The shared representation extracts low-level features that are universal across environments (e.g., edge detection in vision-based RL or generalized inertial dynamics), while the personalized heads map those features to actions tailored to the local environment's specific physics. Similarly, Jang et al. utilize personalized federated hypernetworks to generate custom weights for local agents, allowing rapid adaptation to new environments.
3. Targeting Mixture Environments and Momentum Tracking
Rather than treating environmental variations as noise to be averaged out, modern frameworks reframe the global objective around a mixture environment. As explored by Zhu et al. and Zhu et al., this approach optimizes the global policy to balance the actual distribution of heterogeneous dynamics present across the entire federation, rather than optimizing for a fictional, uniform average environment.
To stabilize training under this objective, Wang et al. introduced momentum mechanisms (such as FedSVRPG-M) into the policy gradient aggregation step. Momentum acts as a low-pass filter on the aggregated updates, smoothing out the high-variance directional conflicts caused by heterogeneous local transitions and ensuring steady progress toward the mixture optimum.
4. Weight-Free Policy Distillation
When agents operate in highly distinct environments, they may require entirely different neural network architectures to function effectively. In such scenarios, standard weight aggregation (like FedAvg) is impossible.
To bypass this, the FedHPD framework developed by Jiang et al. utilizes federated policy distillation. Instead of sharing raw model weights, agents run their local policies on a synthetic public dataset and share only their action probability distributions. The central server aggregates these probabilities, creating a global "teacher" policy that guides local "student" networks without ever needing architectural homogeneity or direct weight blending.
Preserving the Linear Speedup
The ultimate goal of any federated learning system is to achieve a linear speedup: if you scale from 1 agent to $N$ agents, the system should learn $N$ times faster. In supervised federated learning, this speedup is well-established, but in heterogeneous RL, proving that a linear speedup can survive model mismatch has been a significant theoretical hurdle.
Recent theoretical breakthroughs have finally provided these guarantees across different RL paradigms:
- On-Policy Control: Zhang et al. proved that on-policy control algorithms (like FedSARSA) can preserve a linear speedup, demonstrating that $N$ agents can bound their suboptimality error $N$ times faster, provided the environmental divergence remains within a bounded threshold.
- Value Function Estimation: In temporal difference learning, Wang et al. and Beikmohammadi et al. showed that collaborative value estimation under model mismatch still yields a $\sqrt{N}$ to linear reduction in estimation bias and convergence time, proving that collaboration remains highly beneficial even when local environments are permanently mismatched.
- Model-Based Exploration: For exploration-heavy settings, Tiapkin et al. introduced Fed-UCBVI, proving that model-based federated algorithms can achieve near-optimal regret bounds with a linear speedup while maintaining low communication overhead.
These theoretical insights confirm that environmental heterogeneity is not an insurmountable barrier to distributed intelligence. By shifting from rigid global uniformity to flexible, variance-aware aggregation mechanisms, fleets of diverse agents can safely exploit their collective experiences to master complex, real-world environments.
Key Foundational Work
Federated Reinforcement Learning with Environment Heterogeneity (2022) — contributed 5 concept elements to this direction.
Communication efficient parallel reinforcement learning (2021) — contributed 2 concept elements to this direction.
Federated Reinforcement Learning: Linear Speedup Under Markovian Sampling (2022) — contributed 2 concept elements to this direction.