Personalized Rollout Allocation for Federated LQR Policy Optimization under Heterogeneous Process Noise

Mohammadreza Rostami, Shahriar Talebi, Solmaz S. KiaarXiv 2026 · arXiv

TL;DR

In federated control, a few "noisy" clients can easily destabilize the entire fleet's learning process. This paper introduces a personalized rollout allocation strategy that identifies these high-variance clients and assigns them more trajectory rollouts, reducing the one-step instability rate from 37% to just 0.3% under the same communication budget.

What Problem Does This Paper Address?

Federated learning for control—specifically for the Linear Quadratic Regulator (LQR)—aims to let a fleet of robots or devices learn a common, optimal control policy without sharing their raw data. In a "model-free" setting, these clients don't know their own physics; they learn by trying out slightly different control gains (rollouts) and measuring the resulting costs.

The challenge arises from heterogeneous process noise. Imagine a fleet of drones: some are flying in calm indoor air, while others are battling gusty winds. The "windy" drones produce extremely noisy gradient estimates. In standard federated learning, every client performs the same number of rollouts. When the server averages these updates, the high-variance noise from a few "windy" drones can drown out the useful signal from the rest, leading to updates that make the controller unstable and cause the robots to crash.

Furthermore, uplink communication is often a bottleneck. Sending full gradient matrices (which grow with the number of parameters) is expensive.

Methods

The authors propose two main innovations to make federated LQR more robust and efficient:

  1. Scalar Communication Scheme: Instead of sending a full matrix of gradient updates, clients send a single scalar coefficient per rollout pair. The server and clients share a random seed, allowing the server to "regenerate" the random perturbation directions used by the clients. This turns the communication budget directly into a "rollout budget."
  2. Personalized Rollout Allocation: Rather than giving every client the same number of rollouts, the server uses a greedy integer allocation rule (Algorithm 2). It estimates the variance of each client's feedback and assigns more rollouts to the noisier clients.

The core idea is effectiveness: the server calculates how much a single additional rollout from Client A would reduce the total fleet variance compared to a rollout from Client B. Surprisingly, this means the noisiest clients get the most work, effectively "smoothing out" their contribution so they don't derail the collective learning.

Fig. 1. One round of the protocol. The server broadcasts the gain Kk and the rollout counts (dashed). Each client executes its assigned rollout pairs on its own system and uploads one scalar coefficient per pair (solid), subject to a fleet budget of S scalars per round. The server regenerates the perturbation directions from the shared seed, reconstructs the gradient estimate, updates the gain, and reallocates the next round’s counts from the variance of the received coefficients, so that clients with noisier measurements are assigned more rollout pairs.

Datasets

The paper evaluates the approach using simulated trajectory rollouts of a 4-dimensional linear system (resulting in 16 controller parameters).

  • Fleet Size: Tests were conducted with fleets of 10 and 20 clients.
  • Heterogeneity: Noise levels were varied across the fleet, with some clients experiencing up to 40x more noise than others to simulate extreme real-world conditions.

Key Insights and Ideas

  • The "Aha" Moment on Allocation: In many federated settings, you might want to down-weight noisy clients. However, in LQR, every client contributes equally to the fleet objective. You can't just ignore the noisy ones. The "aha" insight is that the only way to fix a noisy client's contribution without biasing the result is to give them more samples.
  • Variance-Optimal Rule: The authors prove that the optimal number of rollouts for a client is proportional to the standard deviation of its estimates ($\sqrt{V_i}$).
  • Stability as a Resource: The paper links rollout allocation directly to control stability. By reducing update variance, they effectively enlarge the "certified step size"—the speed at which the fleet can safely learn without risking instability.
  • Zero-Overhead Estimation: The server doesn't need extra metadata to decide the allocation; it can estimate the required variances directly from the scalar coefficients it already receives.

Key Results

The personalized allocation scheme significantly outperforms uniform allocation across all metrics:

  • Stability: In a fleet of 10 clients with high noise heterogeneity, uniform allocation resulted in a 37.4% instability rate (crashes). Personalized allocation reduced this to 0.3% using the exact same total number of rollouts.
  • Learning Floor: As learning progresses, updates eventually settle into an "error floor" due to sampling noise. Personalized allocation reached an error floor 6x lower than uniform allocation.
  • Efficiency: To reach a specific accuracy target, personalized allocation requires a budget that is $\Gamma$ times smaller (where $\Gamma$ is a factor determined by noise dispersion) than uniform allocation.
Comparison of unstable updates vs. budget and step size. Personalized allocation (blue) stays stable much longer than uniform (red).

Limitations and Future Work

  • Shared Dynamics: The current model assumes all clients share the same underlying physics ($A$ and $B$ matrices). Extending this to clients with different dynamics is a natural next step.
  • Conservative Bounds: The stability certificates are based on second-moment bounds (Markov/Chebyshev-style), which are mathematically rigorous but often conservative in practice.
  • Energy Constraints: The paper assumes the server can assign as many rollouts as it wants to a client (up to the total budget). In reality, a very noisy client might run out of battery if asked to do too many rollouts. Future work could include a "per-client cap" on rollouts.

Summary updated

Abstract

We study federated model-free learning of a common linear quadratic regulator (LQR) gain under a limited uplink communication budget, with clients that share dynamics and costs but experience different process-noise levels. Conventionally, clients transmit gradient matrices whose dimension equals the number of controller parameters. Exploiting the structure of two-point zeroth-order estimates, we consider an implementation in which clients transmit only one scalar cost-difference coefficient per rollout pair, while the server regenerates the associated perturbation directions from shared seeds. Communication thus scales with the number of rollout pairs rather than the controller dimension, turning the uplink budget into a rollout budget. Because noise heterogeneity produces gradient estimates of widely differing variance and every client carries equal weight in the fleet objective, uniform rollout allocation allows noisy clients to dominate the aggregated update and increases the probability of destabilizing updates. We propose a rollout-allocation scheme that assigns rollout pairs in proportion to the standard deviation of each client's estimates and show that it minimizes the variance of the aggregated update. The scheme is implemented exactly through a greedy integer allocation rule whose weights are estimated from the communicated scalars. We further prove that a single allocation factor quantifies its improvement over uniform allocation in a one-round instability bound, the certified step size, and the error floor of a joint stability and convergence guarantee. Simulations show that, under the same communication budget, personalized allocation reduces the one-step instability rate from $37\%$ to $0.3\%$.
BibTeX
@misc{rostami2026personalized,
  author = {Mohammadreza Rostami and Shahriar Talebi and Solmaz S. Kia},
  title = {{Personalized Rollout Allocation for Federated LQR Policy Optimization under Heterogeneous Process Noise}},
  year = {2026},
  eprint = {2611.00712},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00712},
}
Figure