TL;DR
In federated control, a few "noisy" clients can easily destabilize the entire fleet's learning process. This paper introduces a personalized rollout allocation strategy that identifies these high-variance clients and assigns them more trajectory rollouts, reducing the one-step instability rate from 37% to just 0.3% under the same communication budget.
What Problem Does This Paper Address?
Federated learning for control—specifically for the Linear Quadratic Regulator (LQR)—aims to let a fleet of robots or devices learn a common, optimal control policy without sharing their raw data. In a "model-free" setting, these clients don't know their own physics; they learn by trying out slightly different control gains (rollouts) and measuring the resulting costs.
The challenge arises from heterogeneous process noise. Imagine a fleet of drones: some are flying in calm indoor air, while others are battling gusty winds. The "windy" drones produce extremely noisy gradient estimates. In standard federated learning, every client performs the same number of rollouts. When the server averages these updates, the high-variance noise from a few "windy" drones can drown out the useful signal from the rest, leading to updates that make the controller unstable and cause the robots to crash.
Furthermore, uplink communication is often a bottleneck. Sending full gradient matrices (which grow with the number of parameters) is expensive.
Methods
The authors propose two main innovations to make federated LQR more robust and efficient:
- Scalar Communication Scheme: Instead of sending a full matrix of gradient updates, clients send a single scalar coefficient per rollout pair. The server and clients share a random seed, allowing the server to "regenerate" the random perturbation directions used by the clients. This turns the communication budget directly into a "rollout budget."
- Personalized Rollout Allocation: Rather than giving every client the same number of rollouts, the server uses a greedy integer allocation rule (Algorithm 2). It estimates the variance of each client's feedback and assigns more rollouts to the noisier clients.
The core idea is effectiveness: the server calculates how much a single additional rollout from Client A would reduce the total fleet variance compared to a rollout from Client B. Surprisingly, this means the noisiest clients get the most work, effectively "smoothing out" their contribution so they don't derail the collective learning.
Datasets
The paper evaluates the approach using simulated trajectory rollouts of a 4-dimensional linear system (resulting in 16 controller parameters).
- Fleet Size: Tests were conducted with fleets of 10 and 20 clients.
- Heterogeneity: Noise levels were varied across the fleet, with some clients experiencing up to 40x more noise than others to simulate extreme real-world conditions.
Key Insights and Ideas
- The "Aha" Moment on Allocation: In many federated settings, you might want to down-weight noisy clients. However, in LQR, every client contributes equally to the fleet objective. You can't just ignore the noisy ones. The "aha" insight is that the only way to fix a noisy client's contribution without biasing the result is to give them more samples.
- Variance-Optimal Rule: The authors prove that the optimal number of rollouts for a client is proportional to the standard deviation of its estimates ($\sqrt{V_i}$).
- Stability as a Resource: The paper links rollout allocation directly to control stability. By reducing update variance, they effectively enlarge the "certified step size"—the speed at which the fleet can safely learn without risking instability.
- Zero-Overhead Estimation: The server doesn't need extra metadata to decide the allocation; it can estimate the required variances directly from the scalar coefficients it already receives.
Key Results
The personalized allocation scheme significantly outperforms uniform allocation across all metrics:
- Stability: In a fleet of 10 clients with high noise heterogeneity, uniform allocation resulted in a 37.4% instability rate (crashes). Personalized allocation reduced this to 0.3% using the exact same total number of rollouts.
- Learning Floor: As learning progresses, updates eventually settle into an "error floor" due to sampling noise. Personalized allocation reached an error floor 6x lower than uniform allocation.
- Efficiency: To reach a specific accuracy target, personalized allocation requires a budget that is $\Gamma$ times smaller (where $\Gamma$ is a factor determined by noise dispersion) than uniform allocation.
Limitations and Future Work
- Shared Dynamics: The current model assumes all clients share the same underlying physics ($A$ and $B$ matrices). Extending this to clients with different dynamics is a natural next step.
- Conservative Bounds: The stability certificates are based on second-moment bounds (Markov/Chebyshev-style), which are mathematically rigorous but often conservative in practice.
- Energy Constraints: The paper assumes the server can assign as many rollouts as it wants to a client (up to the total budget). In reality, a very noisy client might run out of battery if asked to do too many rollouts. Future work could include a "per-client cap" on rollouts.
Summary updated