Robust reinforcement learning trains a policy against a simulated "nature" that is allowed to choose bad physics. The answer depends heavily on one modeling choice that is usually made for mathematical convenience: how freely nature may change its mind. Under state-action rectangularity, the standard assumption, nature picks a worst-case transition independently at every state and action. A robot joint can be loose in one state and jammed in the next. The result is well-documented over-pessimism. At the other extreme, domain randomization fixes one random set of physical parameters for a whole episode. Real simulator errors usually sit between the two. Friction changes from one patch of floor to the next, motors drift as they heat up, and contact errors are tied to particular configurations.
We propose to treat the coupling scale of the adversary as a measurable quantity, meaning how long or how far its perturbations must stay consistent. It becomes a dial to set, not an assumption to accept. The claim has two parts. First, the transfer error of a robustly trained policy is U-shaped in this coupling scale, with the minimum near the coupling of the actual model error on the target system. Second, that coupling can be estimated cheaply from a few target rollouts. Sweeping the dial on deep continuous control is only affordable because accelerator-native simulators run millions of physics steps per second. They let us evaluate many structured adversaries per policy update and repeat whole training runs at many coupling settings.
Core Claim
Existing practice splits the coupling spectrum between two communities that rarely compare against each other on equal footing. Rectangular robust methods lose performance by preparing for impossible combinations of failures. This "lightning strikes everywhere" problem is analyzed by Derman et al., Tang et al. and Kumar et al., and those papers motivate this proposal. The episode-fixed latent-parameter view of domain randomization, formalized by Chen et al., proves its transfer bounds only when the real system lies inside the randomized family. That limits what it can guarantee.
Concretely, let $\delta_t \in \mathbb{R}^d$ be the perturbation to $d$ physical parameters (masses, friction, motor gains, actuation delay) applied at step $t$ of a horizon-$H$ episode. We define a family of uncertainty sets indexed by a coupling length $\kappa$:
$$ \mathcal{U}_\kappa = \Big\{ \delta_{1:H} : \|\delta_t\|_\infty \le \beta,\ \ \|\delta_{t+1}-\delta_t\|_\infty \le 2\beta/\kappa \Big\} $$
Here $\beta$ is the perturbation radius. At $\kappa = 1$, nature may jump anywhere between steps, which is the time-indexed analogue of rectangularity. As $\kappa \to \infty$, the perturbation is frozen for the episode, which matches the coupling of domain randomization and of parameter-level worst-case methods such as M2TD3. A spatial variant replaces time with distance in state space: $\delta$ becomes a field over states with a Lipschitz bound in place of the temporal bound.
We are honest about the correspondence. Per-step independence matches $(s,a)$-rectangularity only when trajectories rarely revisit states. The spatial variant is the closer analogue, and we report both.
The prediction is that the coupling length minimizing transfer error tracks the estimated coupling length of the target's model error. Training at the wrong end of the dial loses measurably in either direction.
Research Program
The build has three pieces, all written in JAX on top of Brax.
The adversary. We represent members of $\mathcal{U}_\kappa$ with a low-dimensional basis: smooth temporal splines with knot spacing $\kappa$, or radial fields over a state embedding for the spatial variant. Each perturbation is projected onto the constraint set.
The worst-case search. For each policy update we draw $K$ candidate perturbations per environment and evaluate them in parallel across thousands of environment instances. We then run a few steps of zeroth-order ascent on the adversary coefficients, and train the policy (PPO or SAC from Brax) on the worst-scoring fraction. This is a sampled, constrained adversary, not an exact robust Bellman solver.
A low-dimensional coupled adversary should suffice because of Kumar et al. and Kumar et al.. Those papers prove that the worst case over globally coupled $L_p$ transition sets is a rank-one perturbation of the nominal dynamics, with tractable bisection ($L_1$) and spectral ($L_2$) solvers on tabular MDPs. We use that result in two ways. It tells us coupled worst cases are structurally simple, which is why we parameterize the adversary with few coefficients. It also gives us a ground-truth check, described below.
The coupling estimator. We collect a small number of target rollouts, replay their actions in the training simulator, and fit the per-step residual in the perturbation parameters $\delta$ that best explains the mismatch. The estimated coupling length is the decay length of the residual's autocorrelation along the trajectory, or across nearby states for the spatial variant.
The program then asks whether choosing $\kappa$ by this estimator matches the best $\kappa$ found in hindsight by sweeping. A secondary question comes from Chen et al.. They prove memory is necessary under episode-fixed latent parameters. Rectangular theory, by contrast, makes memoryless policies sufficient. We therefore train recurrent and feedforward policies at every $\kappa$ and test the predicted pattern: memory should help more as coupling grows, and help little at $\kappa = 1$. The prior work names this as an open gap but leaves it untested.
Baselines
All methods share the same simulator, the same environment-step budget, the same policy architecture class, and the same hyperparameter search budget. That budget is a fixed number of trials per method, chosen on nominal-environment validation only, never on the target. We report results at 5 million steps to match the RRLS protocol of 10 seeds, and at a larger budget that Brax makes cheap.
Do nothing. The naive reference is nominal training: PPO or SAC on unperturbed physics. Any robust method that fails to beat it on the target is not worth its cost.
Domain randomization. This is the strongest competitor. It samples $\delta$ uniformly from the $\beta$-box once per episode, in both memoryless and recurrent versions. RRLS reports it as the best method for average-case return across the uncertainty set, and Isaac Gym used it for zero-shot real transfer on ANYmal and TriFinger. The file gives no number for it: no reported result found on Lacuna for domain-randomization returns on the RRLS tasks, since RRLS's findings are summarized only qualitatively. We therefore reproduce it ourselves. Beating it means a statistically significant improvement, over 10 seeds, in mean return on the target. We also require no loss of more than a few percent of nominal return, because RRLS shows that worst-case methods tend to buy robustness by giving up nominal performance.
M2TD3 and RARL from RRLS. These represent the two existing ends of the dial. M2TD3 is an episode-fixed worst case over parameters, near $\kappa = \infty$. RARL applies adversarial forces at every step, near $\kappa = 1$. Both are run through the RRLS code (SuReLI/RRLS) on the same perturbation dimensions.
Ablations. Two ablations isolate the key ingredient. The first holds the adversary search, radius and step budget fixed and varies only $\kappa$. If the extremes match the matched $\kappa$, the coupling claim fails. The second is correlated domain randomization: random, non-adversarial perturbations drawn from $\mathcal{U}_\kappa$ at the same $\kappa$. It separates "coupling matters" from "worst-case search matters."
Why not just…?
Why not just use domain randomization with wider ranges and a recurrent policy? This is the most serious threat. RRLS found it beats adversarial methods on average return, and Isaac Gym shows it transfers to real robots. The weakness is structural. Its guarantees from Chen et al. assume the target lies within the randomized episode-fixed family, and Chen et al. show that unrecoverable states make the gap grow linearly with horizon. Our test targets are built to violate the episode-fixed assumption: drifting gains, patchy friction, and the Brax-to-MuJoCo contact-model change. If wide, memory-equipped randomization still matches the matched-$\kappa$ method there, we will report that the coupling dial is unnecessary.
Why not just use standard rectangular robust RL, which keeps clean dynamic programming? Rectangular methods are cheaper per update, as Panaganti et al. note through decomposition into local problems. However, the evidence for over-conservatism is direct. In Rickard et al., an interval-MDP rectangular baseline reached a specification-satisfaction probability of 0.065 on UAV navigation under wind, against 0.195 for a parameter-coupled model. Our $\kappa = 1$ arm, together with RARL, measures whether this collapse carries over to continuous locomotion.
Why not just port the exact $L_p$ dual solvers of Kumar et al. to the GPU? Their solvers need an explicit transition matrix and occupancy measure. Continuous rigid-body control provides neither, and their theory also needs a small radius so perturbed transitions stay valid probabilities. Porting them would therefore mean discretizing, and the resulting set would not describe physical parameter error. We instead use their solvers as a correctness check, which is where they are exact.
Why not just use unsupervised environment design? Regret-driven environment generators in minimax run on grid mazes. They put no constraint on how the generated dynamics are coupled, and unconstrained adversaries "move the goalposts" in RRLS. A regret-based teacher constrained to $\mathcal{U}_\kappa$ is a reasonable later variant, but it does not answer the coupling question.
Proposed Experiments
Validity check. On a discretized pendulum and small grid tasks of a few hundred states, we compare the worst-case value found by our sampled spline adversary with the exact $L_2$ non-rectangular value from the spectral method of Kumar et al.. That paper reports roughly 0.17 seconds at $|S| = 1000$, against over 300 seconds for SLSQP. If our adversary is not within a small margin of the exact value, the deep results are uninterpretable and the search must be fixed first.
Main experiment. We train in Brax on Ant, HalfCheetah, Hopper, Walker2d, InvertedPendulum and HumanoidStandUp. We sweep $\kappa$ over a geometric grid from 1 to episode-fixed, in temporal and spatial variants, with feedforward and recurrent policies. Each policy is evaluated zero-shot on three kinds of target:
- the corresponding RRLS MuJoCo environments with their 1D–3D parameter shifts, which are the episode-fixed targets;
- MuJoCo variants with slowly drifting motor gains and piecewise friction patches, which are the intermediate-coupling targets;
- the unperturbed MuJoCo models, whose contact modeling differs from Brax's spring-based joints and so supplies a gap not contained in any randomized family.
Metrics are mean return, tenth-percentile return across target perturbations, and nominal return. Each is reported with 10-seed intervals, along with wall-clock time.
Decisive checks. The proposal succeeds only if all three of the following hold:
- The return-versus-$\kappa$ curve has an interior or endpoint minimum that shifts with the target's true coupling.
- The $\kappa$ chosen by the estimator, from at most a few dozen target rollouts, falls within one grid step of the best $\kappa$ in hindsight on most tasks.
- At that $\kappa$, the method beats both uniform and correlated domain randomization on the intermediate-coupling and cross-simulator targets.
It must do so without losing nominal return. If correlated randomization matches the adversarial version, we will conclude that coupling, not worst-case search, carries the benefit, which is still a useful and falsifiable finding. The memory analysis is read off the same sweep.
References
- A Bayesian Approach to Robust Reinforcement Learning
- Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization
- Non-rectangular Robust MDPs with Normed Uncertainty Sets
- Understanding Domain Randomization for Sim-to-real Transfer
- Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation
- Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processes
- Rrls: Robust reinforcement learning suite
- Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning
- Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
- Learning robust policies for uncertain parametric Markov decision processes
- minimax: Efficient Baselines for Autocurricula in JAX