Randomizing Physics by How It Changes Behavior

When a robot policy is trained in simulation and deployed on real hardware without any real-world fine-tuning, the standard safeguard is domain randomization: simulator physics such as friction, masses and motor delays are resampled every episode, so that the real robot looks like one more sample. The hard part is deciding how wide to make these ranges. Current automatic methods widen the ranges to maximize their spread measured in raw parameter units. That measure is arbitrary. Doubling a link mass may change nothing the robot does, while a small shift in friction across the stick–slip boundary changes everything. Worse, the answer depends on whether mass is written in kilograms or on a log scale.

We propose to measure the width of a randomization distribution by the behavior it produces instead. Two simulator settings are far apart when the current policy's state–action visits under them are far apart in optimal-transport (Wasserstein) distance. Optimal transport measures how much probability mass must be moved, and how far, to turn one distribution into another. The randomization curriculum then widens the distribution as fast as possible in this behavioral geometry while keeping task success above a threshold. In two sentences: stop counting parameter volume, and count how much distinct physical behavior the training distribution forces the policy to handle. Because this distance does not depend on how the simulator is parameterized, the curriculum becomes invariant to units and reparameterization, which no current method is.

Core Claim

The closest method, Domain Randomization via Entropy Maximization (DORAEMON), maximizes the Shannon entropy $\mathcal{H}(p)$ of a distribution $p(\xi)$ over simulator parameters $\xi$, subject to success rate $\ge \alpha$. It reaches 60% zero-shot success on a real Franka PandaPush task with 17 randomized parameters. We keep its constrained structure and its success constraint. We replace the objective with a behavioral entropy.

Let $g_\phi(\xi) \in \mathbb{R}^k$ be a learned embedding of simulator parameters. It is trained so that $\|g_\phi(\xi) - g_\phi(\xi')\|$ approximates the translation-invariant Wasserstein distance between the policy's occupancy measures (the distributions of visited states and actions) under $\xi$ and $\xi'$. Let $G(\xi) = J_\xi^\top J_\xi + \epsilon I$ be the metric this embedding induces on parameter space, where $J_\xi$ is the Jacobian of $g_\phi$ and $\epsilon$ is a small floor. Entropy measured against this metric's volume is

$$\mathcal{H}_G(p) = \mathcal{H}(p) + \tfrac{1}{2}\,\mathbb{E}_{\xi\sim p}\left[\log\det G(\xi)\right].$$

This follows from the change-of-variables formula for densities on a Riemannian manifold. Any smooth reparameterization of $\xi$ leaves $\mathcal{H}_G$ unchanged, whereas $\mathcal{H}(p)$ shifts by a Jacobian term. We also replace the step-size limit on distribution updates. DORAEMON bounds the KL divergence between successive parameter distributions. We bound the Wasserstein distance between successive distributions under the same ground metric $G$, which caps how much new behavior each curriculum step introduces.

The claim to test is that this geometry concentrates randomization effort where physics actually changes behavior. That should yield better zero-shot transfer at equal sample budget, and results that do not change under reparameterization.

How This Differs From Existing Work

Three lines of prior work motivate the idea without doing it.

Generalization through Diversity (DIPLR) already uses Wasserstein distance between agent occupancy measures to measure diversity. It does so to choose among discrete generated levels held in a replay buffer, by combining regret with diversity. It never shapes a continuous distribution over physics parameters, and it has no success-constrained expansion. Our contribution is to turn that pairwise distance into a metric on parameter space and use it inside a constrained distribution update. Because DIPLR's approach could be ported directly to sampled physics parameters, we run exactly that port as a baseline.

Active Domain Randomization (ADR) rewards a particle-based sampler with a binary discriminator that separates rollouts in $\xi$ from rollouts in a fixed reference environment. That reward can saturate when behaviors stop overlapping, and it depends on the choice of reference. We use no reference environment and no discriminator.

DORAEMON supplies the constrained entropy-maximization template we modify.

The method draws on two sources.

Kantorovich–Rubinstein-style occupancy comparison comes from Wasserstein Distance Maximizing Intrinsic Control and the occupancy-trajectory analyses in Studying Exploration in RL. We use their evidence that transport distances keep giving a signal where overlap-based divergences go flat. We apply that distance to environments, not to skills or to policy iterates.

The translation-invariant distance comes from Relative-Translation Invariant Wasserstein Distance. That paper proves $W_2^2 = \|\bar\mu-\bar\nu\|^2 + RW_2^2$, splitting the distance into a shift of the means and a change of shape. We use the shape part $RW_2$. Otherwise a walker that simply travels farther under lower friction would look "behaviorally different" even though its gait is unchanged.

Hyperparameter Selection for Imitation Learning supports choosing state-distribution distances over action-space proxies. In that paper, Wasserstein distance between visited-state distributions tracked true performance, while action mean-squared error did not. The same paper also warns that the choice of ground metric matters, which constrains us to proprioceptive state for now.

Research Program

We build the method on top of DORAEMON's public formulation, using a factorized Beta distribution over normalized parameters.

Each iteration begins by reusing the rollouts already collected for policy training. Each rollout is tagged with the $\xi$ that generated it. We then compute Sinkhorn estimates of $RW_2$, following the pre-centering algorithm of the $RW_2$ paper, between state-occupancy samples for a minibatch of $\xi$ pairs.

Next we fit $g_\phi$ by regressing embedding distances onto these transport distances. This amortizes the pairwise cost, so the number of optimal-transport solves per iteration stays fixed rather than growing quadratically.

Finally, the distribution update takes a gradient step on $\mathcal{H}_G(p)$, subject to two constraints: the success constraint, estimated as in DORAEMON, and a behavioral trust region on the step size.

Because $g_\phi$ is re-fit as the policy improves, the geometry follows the current policy. A parameter that stops mattering once the policy learns to compensate for it becomes cheap to widen.

We will also analyze what the learned metric discovers. On PandaPush this means checking whether the metric's largest directions line up with box friction and center of mass, rather than joint damping. On locomotion it means checking whether $G$ spikes near known contact-regime transitions.

Baselines

The naive reference points are Uniform Domain Randomization over the widest plausible hypercube and nominal-only training. DORAEMON reports that fixed-wide randomization reaches 0% real PandaPush success, so we expect it to set the floor.

The strongest existing method is DORAEMON. Its reported real-world PandaPush result is 60% success within 3 cm, with 2.68 cm mean final distance. In the same paper, LSDR reaches 46.7% and AutoDR reaches 26.7%. We run DORAEMON with its released settings, the same success threshold $\alpha$, the same policy learner, the same environment-step budget and the same tuning budget. We count as beating it any statistically reliable improvement in held-out-dynamics success at equal steps. On hardware, the bar is exceeding 60% under the same protocol.

ADR is run with its discriminator-rewarded particle sampler. We also run a variant whose discriminator is replaced by a Wasserstein critic with gradient penalty. That variant isolates whether a smoother reward alone closes the gap.

DIPLR-continuous is the direct port of DIPLR. It samples candidate $\xi$ into a level buffer and replays them by regret plus occupancy-Wasserstein diversity, with the same distance estimator we use.

Two ablations isolate the key ingredients:

  • Sensitivity-DORAEMON. DORAEMON with parameters rescaled by finite-difference sensitivity of scalar return. This tests whether occupancy geometry is needed at all.
  • Standard $W_2$ in place of $RW_2$. This tests whether translation invariance matters.

All methods share seeds (10, following the RRLS protocol) and the same environment-step budget, and all sampler overhead is counted in wall-clock time.

Why not just…?

Why not just use DORAEMON with sensibly normalized parameter ranges? Normalization fixes units, but it cannot fix a policy-dependent, nonlinear sensitivity such as a friction threshold whose importance changes as the policy learns. The reparameterization experiment below tests this directly. If DORAEMON's results stay stable under log-versus-linear rescaling and the sensitivity-rescaled variant matches our method, then the transport machinery is unnecessary. We would report that and fall back to the cheaper method.

Why not just run DIPLR on sampled physics parameters? This is the most serious threat, because it uses the same distance. The difference lies in what gets optimized. A replay buffer reweights samples that have already been drawn. It cannot widen the support of the distribution under a success constraint, so it cannot find the edges of the solvable region. If DIPLR-continuous matches us on held-out dynamics, the contribution shrinks to the constrained-expansion machinery. The head-to-head comparison settles this.

Why not just swap ADR's discriminator for a Wasserstein critic? It is a one-line change and we include it. It still measures distance from a single reference environment, so it rewards being different from the default rather than covering distinct behaviors. We expect it to chase one hard region. We measure this with the spread of the particle sampler across behavioral clusters.

Proposed Experiments

The primary testbed is RRLS, covering Hopper, Walker2d, HalfCheetah and Ant with 1D to 3D uncertainty sets, extended to 8 or more parameters. Evaluation is sim-to-sim. Policies are tested on a held-out "reality" set drawn from outside the training bounds and perturbed by unmodeled effects, specifically action latency and actuator noise. Metrics are mean and worst-decile return across held-out dynamics. No reported numeric RRLS results were found on Lacuna, so all numbers come from our own reruns.

The secondary testbed is LunarLander with engine-strength sweeps. Here ADR showed uniform randomization failing at low strength, but no reported numeric result was found on Lacuna.

If hardware is available, the transfer test is PandaPush, compared against DORAEMON's 60%.

Two checks decide whether the idea holds.

The first is the reparameterization test. We rerun every method with parameters expressed in log, linear and randomly warped monotone coordinates. The prediction is that our method's held-out return varies within seed noise, while DORAEMON's and ADR's shift. If ours also shifts, the invariance claim fails.

The second is the sample-efficiency test at matched steps against DORAEMON and DIPLR-continuous. It is paired with the $RW_2$-versus-$W_2$ ablation on locomotion, where translation-driven differences should otherwise dominate the distance. If neither check favors the behavioral geometry, the conclusion is that parameter-space entropy was already adequate.