Balanced-Hardness Physics Randomization Against Robust Overfitting in Sim-to-Real Control

Robots are usually trained in simulation and then deployed on hardware without any real-world tuning. The standard trick for making that jump is domain randomization: during training, the simulator's physics (masses, friction, motor delays) and appearance are resampled at random, so the policy learns to cope with many possible worlds. A natural upgrade is to make the randomization adversarial, so that the simulator deliberately picks the physics the current policy handles worst. In image classification, training against worst-case inputs has a well-documented side effect called robust overfitting. Robustness on held-out data peaks early and then declines while training loss keeps falling, because the model starts memorizing its specific training perturbations. On CIFAR-10, the robust test error reported in Overfitting in adversarially robust deep learning rises from 43.2% at the best checkpoint to 51.4% at convergence. Nobody has checked whether adversarial physics randomization does the same thing to robot policies. If it does, the usual fix of stopping at the best validation checkpoint is not available, because zero-shot transfer means there is no real-world validation data.

This note proposes to measure that effect and to remove it cheaply. The central bet is a transfer of the main finding of Data augmentation alone can improve adversarial training: augmentations prevent robust overfitting when they are diverse and only moderately hard, and they fail when they are maximally hard. Adversarial physics randomization does the opposite. It concentrates training on a narrow set of hardest simulator settings. We propose a different rule: sample physics parameters so that the policy's expected return falls inside a target band of intermediate difficulty, and spread samples evenly across everything inside that band. The scores needed for this rule come from episodes the parallel simulators are already running, so the method adds essentially no compute over uniform randomization. It never backpropagates through the policy to build perturbations.

Core Claim

We make two linked claims. The first is diagnostic: worst-case physics randomization causes late-stage robust overfitting in continuous control. Return on held-out physics, meaning parameter values outside the training range and parameter types never randomized, peaks and then declines even as return under the training adversary keeps rising. The decline grows with how concentrated the adversary's sampling distribution becomes. The second claim is about the fix. A sampler that targets a band of intermediate hardness, with coverage spread evenly within the band, keeps most of the sample-efficiency benefit of adversarial selection and removes the late decline. It does this at the cost of uniform randomization and without a real-world stopping criterion.

The mechanism follows the "lookup table" account of robust overfitting. On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views and Conserve-Update-Revise argue that high-capacity networks memorize individual training perturbations rather than learn smooth behavior. A physics adversary that keeps returning to the same few brutal friction-and-mass combinations hands the policy exactly such a table to memorize. Those papers supply the motivating mechanism; we test whether it holds in control.

Relation to Prior Work

Three works already combine adversarial selection with domain randomization, and none studies late-training degradation. VADRA treats the simulator as a player that picks rendering parameters to maximize a vision model's loss, using a REINFORCE update. It reports gains over uniform randomization on Syn2Real and VIRAT, and its authors' own analysis flags that the simulator can drift toward degenerate configurations. It motivates our worry, but it covers static images, not control. RAPPO skips simulator parameters and perturbs the agent's observed state toward low-value states. It fights over-conservatism by mixing nominal and adversarial transitions with a global weight $\alpha$ that an upper optimization level tunes. It is the closest competitor and the strongest existing method on our MuJoCo tasks. It pays for its perturbations with value-function gradients at every step and only reports convergence-time results. EASI uses an adversarial discriminator to identify true physics, but it needs a real trajectory, so it is not zero-shot. We use it only as a hardware-side reference.

Two works supply ingredients. Data augmentation alone can improve adversarial training gives us the idea of separating hardness from diversity. It reports end-of-training AutoAttack robustness on CIFAR-10 of 50.74% with its augmentation pipeline (IDBH) versus 48.21% for the early-stopped baseline. Its scope is image classification, and translating its idea into physics distributions is the step we add. Understanding and Improving Fast Adversarial Training constrains the design. Its analysis shows that one-step gradient adversaries collapse unless stabilized by a regularizer that requires double backpropagation. That cost is the reason we avoid gradient-based adversaries entirely, rather than try to make them cheap.

Two works set the context for evaluation. Robust Domain Randomization for Reinforcement Learning shows that widening visual randomization in CarRacing-v0 drives a standard agent to negative returns, while a feature-consistency regularizer reaches 553. Its authors state that this kind of representation invariance is inappropriate for dynamics changes, which is why we target dynamics. Sim-to-Real: Learning Agile Locomotion For Quadruped Robots represents the prevailing assumption that more simulated training under randomization is monotonically good. Our diagnostic experiment tests that assumption directly.

Research Program

The method keeps a running buffer of pairs $(\xi, R)$, where $\xi$ is a vector of simulator physics parameters and $R$ is the episode return the current policy earned under it. Thousands of parallel environments produce these pairs for free. When an environment resets, its new parameters are drawn from the band distribution

$$q(\xi) \propto p_0(\xi)\,\mathbb{1}\left[\,Q_{a} \le \hat R(\xi) \le Q_{b}\,\right],$$

where $p_0$ is the ordinary uniform randomization prior, $\hat R(\xi)$ is a nearest-neighbour estimate of current return from the buffer, and $Q_a, Q_b$ are the $a$-th and $b$-th quantiles of recent returns. A small fixed fraction of draws comes from $p_0$ to keep $\hat R$ calibrated. Setting $a=0$ with a narrow band recovers worst-case selection. Setting the band to all returns recovers uniform randomization. The band position therefore works as a hardness dial and the band's coverage under $p_0$ as a diversity dial, which lets us run the same hardness-versus-diversity sweep that Data augmentation alone can improve adversarial training ran for images. The sampler costs a nearest-neighbour lookup per reset, with no extra policy gradients and no extra rollouts.

To measure robust overfitting, each run trains roughly three times longer than the usual convergence budget and saves dense checkpoints. Every checkpoint is evaluated on three held-out sets, none ever sampled in training. The first holds the randomized parameters at values outside the training range. The second varies parameter types that were never randomized, such as body size, gravity, and actuator latency; RAPPO's unseen-parameter tests follow the same pattern. The third applies observation perturbations. The robust-overfitting gap is defined as best-checkpoint held-out return minus final-checkpoint held-out return. We also track the entropy of the training sampler over time to test whether the gap tracks sampler collapse.

Baselines

The naive reference is uniform domain randomization over hand-set ranges, trained with PPO, together with a variant with doubled ranges. Uniform randomization is what practitioners actually ship, and it is a real threat: if it shows no late decline and matches held-out return, adversarial selection buys nothing on these tasks. Both variants use the same PPO hyperparameters, environment count, and step budget as every other method. Beating them means matching their final held-out return in fewer environment steps, or exceeding it at equal steps.

The strongest existing method is RAPPO (Revisiting Domain Randomization via Relaxed State-Adversarial Policy Optimization), run with its published state-perturbation and $\alpha$-adaptation procedure on HalfCheetah, Hopper, Walker2d, Ant, and Humanoid, using 5 seeds and 50 initial states as in that paper. The file reports that RAPPO had the best average rank against PPO, MRPO, and SCPPO, but gives no per-environment scores; no reported numeric result was found on Lacuna for RAPPO's per-task returns. We will therefore rerun it rather than quote it. RAPPO receives the same environment-step budget and a hyperparameter sweep of equal size. We report wall-clock time separately, since its per-step gradients are the overhead we claim to avoid. Beating RAPPO means a higher final held-out return at equal environment steps and a smaller robust-overfitting gap, at lower wall-clock time.

The worst-case physics adversary is our own machinery with the band pinned to the bottom quantile. It plays the role of a VADRA-style hardest-sample selector for control, and the diagnostic claim rests on it showing a late decline.

The key ablation is a fixed mixture: half uniform draws, half worst-case draws. It isolates whether "intermediate hardness spread evenly" matters, or whether any injection of easy samples into an adversarial stream suffices.

Validation-based early stopping on a held-out simulated physics family is the zero-shot-compatible version of the fix in Overfitting in adversarially robust deep learning. It is applied to the worst-case adversary. Its validation family is disjoint from the test families.

For the visual CarRacing-v0 extension, the regularized agent of Robust Domain Randomization for Reinforcement Learning is the reference. It reports an average return of 553 in the large randomization domain, against 60 for the nominal agent and negative scores for the standard randomized agent.

Why not just…?

Why not just widen the uniform ranges? Wider ranges couple diversity and hardness. The CarRacing-v0 collapse to negative returns reported in Robust Domain Randomization for Reinforcement Learning and the conservatism reported in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots both suggest that widening eventually hurts. This is still the alternative most likely to tie us on easy tasks. The doubled-range baseline settles it: if it matches the band sampler on out-of-range test return, our gain is only sample efficiency, and we will report it that way.

Why not just early-stop on held-out simulated physics? This needs no real data, so the zero-shot objection does not apply to it, and it is a genuine competitor. Two considerations favour the band sampler. In images, early stopping loses to diversity-preserving augmentation (48.21% versus 50.74% AutoAttack robustness in Data augmentation alone can improve adversarial training). Separately, a simulated validation family is one more hand-chosen range that may not predict hardware. The direct comparison on held-out families, plus the hardware check below, decides it. If early stopping ties us everywhere, the contribution reduces to the diagnosis and a cheap stopping recipe, and we will say so.

Why not just mix uniform and adversarial samples at a fixed ratio, or use RAPPO's $\alpha$? Mixing adds easy samples but leaves the adversarial half concentrated on the same few settings, which is exactly the memorization the lookup-table account predicts. RAPPO's global $\alpha$ makes the same easy-plus-hardest trade at the level of state perturbations. The fixed-mixture ablation tests this head-on. If it closes the overfitting gap as well as the band does, the "balanced hardness" ingredient is unnecessary, and we would recommend the simpler mixture.

Proposed Experiments

The diagnostic experiment comes first and stands on its own. Across five MuJoCo tasks and Isaac Gym Ant and Cartpole, with 5 seeds each, does the worst-case adversary show a positive robust-overfitting gap on held-out physics? Does the gap grow with policy width, as the representation-complexity account predicts? Does it coincide with falling sampler entropy? A null result here is informative too, since it would mean the image-classification phenomenon does not carry over to control. That would argue against importing its remedies.

The method experiment sweeps band position and width. Its target is the hardness-versus-diversity surface on final held-out return. We expect a ridge at intermediate hardness and high coverage, with the worst-case corner showing the largest gap. The decisive comparisons are band versus fixed mixture, and band versus early stopping, both on final held-out return. We also compare band versus RAPPO on return per wall-clock hour.

The hardware check uses physical Cartpole, mirroring EASI. EASI reports roughly 37% lower pole-angle error and 25% lower cart-velocity error than uniform randomization, using one real trajectory. We would deploy several checkpoints from the worst-case and band runs zero-shot. The test is whether the late decline seen on held-out simulated physics also appears on hardware. That single correlation decides whether in-simulation held-out families are a usable zero-shot proxy, and so whether either our method or simulated early stopping can be trusted without real data.