How Far Ahead Should a Domain-Randomized Controller Look?

Controllers that plan ahead using forecasts of upcoming disturbances or reference trajectories improve very quickly with forecast length. In linear control with a perfectly known model, each extra step of lookahead shrinks the gap to the best possible controller by a constant factor, so a lookahead that grows only logarithmically with the task length already gives constant excess cost. That guarantee assumes the model is exact. Robots trained in simulation never have an exact model. The usual remedy is domain randomization: train or plan against many simulators whose physical parameters (mass, friction, damping) are drawn from a distribution wide enough to cover the real robot. The two ideas pull against each other. Lookahead pays off only if the model predicts well many steps out, and randomization deliberately admits that the model is wrong by an amount set by the width of the parameter distribution.

This note proposes to quantify that tension and use the result. We conjecture that in a controller planning over an ensemble of randomized models, the lookahead benefit keeps decaying exponentially until model error accumulated along the planning horizon overtakes it. This gives an optimal horizon that grows like the logarithm of the data used to narrow the parameter distribution. Because the horizon rule depends only on quantities available in simulation (the randomization width and the closed-loop contraction rate), it can be set zero-shot, without the real-world tuning that sim-to-real transfer forbids. The same analysis says when extra lookahead, which costs compute, can replace extra identification or training samples, which are the scarce resource.

Core Claim

Consider linear tracking control $x_{t+1} = A(\theta)x_t + B(\theta)u_t + w_t$, where $\theta$ holds the unknown physical parameters and $w_t$ are disturbances forecast $k$ steps ahead. The controller plans over models $\theta$ drawn from a randomization distribution of radius $r$ around an estimate. When $r$ comes from $N$ data points, as in the Fisher-information ellipsoid construction below, $r$ scales as $1/\sqrt{N}$. Our working hypothesis is

$$\text{Regret}(k, r) \;\lesssim\; C_1\,\lambda^{2k}\,T \;+\; C_2\, r^2\, k^2\, T \;+\; C_3,$$

where $T$ is the task length, $\lambda\in(0,1)$ is the closed-loop contraction factor, and $C_1, C_2, C_3$ are constants that depend on the cost and the disturbance scale. The first term is the known lookahead decay from The Power of Predictions in Online Control. The second is our conjecture for the mismatch penalty. In a contracting system, a parameter error of size $r$ injects a bounded state error at each step, so predicted states drift roughly linearly over $k$ steps, and the resulting cost error grows quadratically. Balancing the two terms gives

$$k^\star \approx \frac{\log(1/r)}{\log(1/\lambda)} \approx \frac{\log N}{2\log(1/\lambda)},$$

so the useful horizon grows logarithmically in the identification data. Past $k^\star$, extra lookahead actively hurts. The $k^2$ exponent is a hypothesis. Averaging over the ensemble and replanning with feedback may soften the drift, and pinning down the true exponent is part of the theory work. What we claim as new is a lookahead-dependent regret bound that holds under a randomized model, together with a horizon rule computed from simulation quantities that the literature can then test.

Where This Sits

Three strands supply the pieces, but none combines them. First, lookahead analyses for known dynamics establish the exponential decay. The Power of Predictions in Online Control bounds the dynamic regret of model predictive control (MPC: re-solve a $k$-step plan every step, execute the first action) by $O(\lambda^{2k}T+1)$ for LQR, the linear quadratic regulator, the standard linear-dynamics, quadratic-cost control problem. Online Optimal Control with Linear Dynamics and Predictions obtains $O(\rho^W)$ for cheap receding-horizon gradient methods and shows that the rate depends on how quickly inputs reach states. Online Optimization With Predictions and Switching Costs proves matching lower bounds with only $W+1$ gradient steps per stage. All three assume exact $A$ and $B$. We keep their controllers and drop that assumption. Their perfect-model curves serve as our upper reference.

Second, Domain Randomization is Sample Efficient for Linear Quadratic Control shows that randomizing over a Fisher-information confidence ellipsoid achieves certainty equivalence's optimal $1/N$ excess cost. Certainty equivalence estimates the parameters and acts as if they were exact. Randomization matches that rate while avoiding its unstable burn-in period. That paper supplies our width-versus-data construction. It analyzes a static feedback controller and never varies a planning horizon, and it reports its burn-in bounds as conservative. We ask whether lookahead shortens burn-in further.

Third, adaptive sim-to-real methods motivate the problem without solving it. RMA infers hidden physics from past sensor history and cannot anticipate what is coming. World Models as Reference Trajectories for Rapid Motor Adaptation uses one-step world-model predictions to correct actions, but does not plan several steps ahead and does not train under randomization. Methods that attack the cost of randomized training directly, such as Cyclic Policy Distillation, Distilled Domain Randomization and Active Domain Randomization, change how policies are trained or which parameters are sampled. Here the lever is test-time lookahead. Those methods act as constraints on our claim: if a well-trained randomized policy already closes the gap, lookahead adds nothing.

Research Program

The theory stage proves or refutes the bound above for ensemble MPC on LQR tracking. Ensemble MPC plans the next $k$ steps to minimize average cost across $M$ models drawn from the Fisher ellipsoid of radius $r_N$. It first handles exact disturbance forecasts and then forecasts whose error grows with lead time. The deliverables are the actual mismatch exponent in $k$, how $k^\star$ depends on $r$ and $\lambda$, and whether averaging over the ensemble beats planning with the single nominal model by a constant or by a rate. We also derive the joint budget question. For a target excess cost $\epsilon$, the bound defines which pairs of data size $N$ and horizon $k$ reach $\epsilon$, and the slope of that curve says how many samples one extra step of lookahead is worth.

The algorithm stage turns $k^\star$ into a rule for nonlinear systems. Here the ensemble is simply the simulator run at parameters sampled from the randomization distribution. The planner is either sampling-based MPC or the receding-horizon gradient scheme of Online Optimization With Predictions and Switching Costs. Two quantities are estimated inside simulation: $\lambda$, from how fast perturbed closed-loop rollouts reconverge, and the effective $r$, from how far ensemble members disagree over the horizon. The planner then sets $k$ to the point where predicted mismatch cost overtakes predicted lookahead gain. A state-dependent variant shortens the horizon when ensemble disagreement spikes, for example at contact changes. No real-system data enters the choice of $k$.

Proposed Experiments

Linear experiments reuse the setups of the cited theory papers. One is the figure-eight double-integrator tracking task from The Power of Predictions in Online Control. There, with an exact model, the cost gap to the offline optimum fell "from approximately 1000 (with no prediction) to nearly $10^{-7}$ with just 15 steps of prediction." The other is the 3D linear system and nonlinear pendulum from Domain Randomization is Sample Efficient for Linear Quadratic Control, which reports $1/N$ decay but no reported numeric values on Lacuna. We sweep $N$, which fixes the width $r_N$, and $k \in [0, 30]$, and record dynamic regret against the true-system offline optimum over 20 draws of the true parameters. The predicted signature has three parts: regret curves in $k$ that are U-shaped rather than monotone, minima that shift right by roughly a constant each time $N$ is multiplied by a fixed factor, and flattening onto the known-model exponential curve as $N$ grows.

Nonlinear experiments use the parameter-uncertainty environments of RRLS: robust-invertedpendulum, robust-hopper, robust-halfcheetah and robust-walker2d under its 1D–3D uncertainty sets. Each method plans or trains over the randomization set and is tested on held-out parameters inside and just outside that set. We report mean and worst-case return, following RRLS. We also use the DeepMind Control actuator-perturbation tasks from World Models as Reference Trajectories for Rapid Motor Adaptation, measuring reward recovery after gain shifts. Simulation samples consumed and planner compute per step are logged throughout.

Baselines

The naive reference points come in two forms. One is certainty-equivalent MPC with the fixed long horizon used in the known-model papers. It plans with the point estimate and always looks as far ahead as possible, which is what applying The Power of Predictions in Online Control as written would do. The other is a randomized controller with no lookahead ($k=0$). Beating both at every $N$ is the minimum requirement.

The strongest linear competitor is the gradient-based randomized controller of Domain Randomization is Sample Efficient for Linear Quadratic Control, run with the same Fisher ellipsoid and the same $N$. No reported numeric result was found on Lacuna for its excess cost, so we reproduce it. Beating it means lower excess cost at small $N$ (shorter burn-in) with no loss at large $N$.

On RRLS the strongest competitor is the TD3 domain-randomization baseline, which that suite found "a powerful baseline for average-case performance." No reported numeric result was found on Lacuna for its scores, so we rerun it at the suite's protocol of 5 million steps and 10 seeds. Our planner uses a terminal value from a policy trained with the same sample budget. Winning means matching its mean return while improving held-out worst-case return, or matching both with fewer training samples. On DeepMind Control, the Reflexive World Model baseline reports "94.3% reward recovery" on Walker-Run versus 66.3% for fine-tuning (World Models as Reference Trajectories for Rapid Motor Adaptation). We run it on identical perturbations with a matched online compute budget.

Four ablations isolate the key ingredient. The first plans with a single nominal model at $k^\star$, testing whether the ensemble matters. The second uses a fixed randomization width not tied to $N$. The third fixes $k$ at a sensible default. The fourth is an oracle that grid-searches $k$ on the test system itself; it is an upper bound that zero-shot deployment cannot use.

Why not just…?

Why not just tune the horizon as a hyperparameter? In simulation this is easy, and it is the real threat. The issue is which system you tune on. Grid search on the nominal simulator picks the known-model optimum and, by our hypothesis, overshoots on reality. Grid search on reality is unavailable zero-shot. The oracle ablation settles the question. If the simulation-computed $k^\star$ gets within a small margin of the oracle while nominal-sim tuning does not, the rule earns its place. If nominal-sim tuning already matches the oracle, the mismatch term is negligible in practice, and we will report that.

Why not just use MPC with online system identification? That is certainty equivalence with lookahead. Domain Randomization is Sample Efficient for Linear Quadratic Control shows certainty equivalence has an unstable burn-in, and RMA notes that explicit identification is hard in real time during contact changes. It is included as the naive baseline and swept over $N$, and the low-$N$ regime directly tests whether the ensemble planner avoids its failure.

Why not just train a randomized model-free policy and skip planning? On moderate randomization this may suffice; RRLS finds it strong on average. But Cyclic Policy Distillation reports that SAC with randomization "failed to learn" on wide-range HalfCheetah, and Active Domain Randomization reports high variance on 8-parameter ErgoReacher. The sample-budget comparison on RRLS shows whether short lookahead lets a weaker, cheaper policy match it.

Why not just use worst-case robust MPC? RRLS and the LQR randomization analysis both find minimax methods conservative, with nominal performance lost even as data grows. Ensemble averaging recovers nominal performance as $r_N$ shrinks. The large-$N$ end of the linear sweep checks this directly.

Decisive Checks

The project hinges on three observations. The first is U-shaped regret in $k$ on the linear tasks, with a minimum that moves logarithmically in $N$. If regret is monotone in $k$, the core hypothesis is false and the practical rule is simply "look far." The second is the simulation-computed $k^\star$ landing near the oracle horizon on held-out RRLS parameters. The third is lookahead measurably shortening the burn-in of randomized control at small $N$, which would turn planning compute into saved samples.