Domain Randomization as a Reward-Identification Tool

Robots trained in simulation face two problems that are usually solved separately. Sparse rewards, where the agent learns only "success" or "failure" at the end of a long episode, make learning slow. The reality gap, where a simulator's physics differ from the real robot's, makes the learned behavior fragile. The standard fix for the gap is domain randomization: train across many randomly sampled versions of the physics (masses, friction, latency) so the policy cannot overfit to one. This proposal argues that domain randomization is also exactly what is needed to learn a reward that does not depend on the physics. Seeing the same task demonstrated under many different physics settings is what lets a reward learner tell "what the demonstrator wanted" apart from "how this particular world happened to respond."

Concretely, we learn a dense, state-only reward from a handful of demonstrations collected across a randomized simulator. We keep the reward itself blind to the physics parameters but give an auxiliary "shaping" term access to them. That dense reward then drives sparse-reward manipulation under wide randomization. Because the reward depends only on observable state, it can be carried unchanged onto the real robot, where it supplies a dense signal for fine-tuning without new real demonstrations. In two sentences: inverse reinforcement learning breaks when the simulator's physics change every episode, unless the part of the model that absorbs physics is told which physics it is in. Fix that, and randomization stops being a nuisance for reward learning and becomes the source of its invariance.

Core Claim

Adversarial inverse reinforcement learning (AIRL) recovers a reward by training a discriminator to separate expert transitions from policy transitions (Learning Robust Rewards with Adverserial Inverse Reinforcement Learning). Its key design writes the discriminator's score as a state-only reward plus a potential-difference "shaping" term that soaks up dynamics-specific structure. That paper reports the resulting reward transferred to a Disabled Ant scoring 130.3 where GAIL scored −58.8. Its guarantee assumes one fixed set of dynamics, in which the potential converges to the optimal value function. Under domain randomization the optimal value $V^*(s;\xi)$ depends on the physics sample $\xi$. A potential that sees only $s$ can represent at best an average over $\xi$, and the leftover leaks into the reward. The file flags this tension directly (Gap 3): AIRL has never been trained over a randomized family of simulators.

Our modification conditions only the potential on the privileged physics parameters:

$$f(s,a,s';\xi) = g_\theta(s) + \gamma\, h_\phi(s',\xi) - h_\phi(s,\xi),$$

Here $g_\theta(s)$ is the learned reward, shared across all physics samples. $h_\phi(s,\xi)$ is a shaping potential that receives the simulator's sampled parameters $\xi$, which are known in simulation, and $\gamma$ is the discount factor. This borrows the privileged-information trick of Asymmetric Actor Critic for Image-Based Robot Learning, which gives the critic full simulator state while the actor sees only robot observations. We apply that asymmetry to the reward learner instead of the critic.

The claim is threefold. First, $g_\theta$ learned this way is more invariant to physics than state-only AIRL trained over the same randomization. Second, its invariance improves as randomization widens, while the difficulty of learning a policy from the sparse reward alone gets worse as randomization widens. Third, optimizing $g_\theta$ solves sparse-reward manipulation under wide randomization and transfers to hardware, both zero-shot and as a fine-tuning reward.

Research Program

The work has three parts.

The first part builds the reward learner. Demonstrations are teleoperated in simulation, with a new $\xi$ drawn per episode from ranges over link mass, friction, joint damping, controller gains, and action latency. These are the parameters used by Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. Human operators adapt naturally to each physics sample, so the demonstrations show one goal reached under many dynamics. That variation is what the learner needs.

The second part trains the policy. A recurrent policy, which infers the physics implicitly from its recent history as in that same paper, is trained under randomization on the sparse success signal plus $g_\theta$. The reward and policy are learned jointly, as in AIRL.

The third part deploys. A zero-shot evaluation on hardware is followed by real-world fine-tuning that uses $g_\theta$ evaluated on estimated object and robot state. Here $g_\theta$ plays the role of a dense progress signal, similar to the latent-distance reward of Accelerating Visual Sparse-Reward Learning with Latent Nearest-Demonstration-Guided Explorations or the pairwise progress critic of A Generalist Pair-wise Progress Critic Model for Vision-Language-Action Robots. Unlike those, $g_\theta$ is learned to be invariant to physics. It also does not assume progress increases monotonically over time, an assumption the VLAC authors identify as a failure source.

This design relates to three other kinds of prior work. State-marginal matching (f-IRL: Inverse Reinforcement Learning via State Marginal Matching) and joint reward-dynamics estimation (Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamics) also aim to separate goals from physics, but they learn in a single source system and test in a single target. Classifier-based correction methods (Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers, Dynamics Adapted Imitation Learning) adjust the reward for a specific shift and require target-domain samples. Our approach needs no target samples for the zero-shot stage. DrEureka: Language Model Guided Sim-To-Real Transfer pairs randomization with language-model-written rewards. Those rewards are heuristic and make no invariance claim. The comparison with DrEureka is reported below as a related system, not a controlled baseline.

The known limitation is the one DARC states: no reward can fix real dynamics that fall outside the simulator's support. We therefore include held-out physics both inside and outside the training ranges.

Baselines

The do-nothing reference is the recipe of Sim-to-Real Transfer of Robotic Control with Dynamics Randomization: a recurrent policy, the same randomization, sparse reward, and hindsight experience replay (HER), which relabels failed episodes with the goals they actually reached. This is the strongest established sparse-reward randomization method in the file. On real Fetch puck pushing it reports 0.89 ± 0.06 success, against 0.67 for a feedforward policy with randomization. On puck pushing we must match it. On harder multi-stage tasks we must beat its success rate at equal environment steps. To make the comparison fair, it receives the same demonstrations, added to its replay buffer, along with the same simulator steps and the same number of hyperparameter trials.

The engineered reference is robosuite's built-in staged dense shaping rewards (robosuite: A Modular Simulation Framework and Benchmark for Robot Learning) under the same randomization. We know of no reported result on Lacuna for shaped-reward Nut Assembly under dynamics randomization. Robosuite and SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark only report that Nut Assembly remains challenging or unsolved.

Two baselines isolate the reward learner. The first is the key ablation: state-only AIRL trained over the identical randomization with a potential $h_\phi(s)$ that does not see $\xi$. It differs from our method only in that one input. The second is GAIL-style imitation under the same conditions. Both receive the same demonstrations and compute.

The fourth comparison is LaNE-style nearest-demonstration latent reward under randomization. It tests whether any demonstration-derived dense reward suffices.

Beating the baselines means two things. First, higher success on held-out physics, averaged over at least five seeds with confidence intervals, at matched simulator steps. Second, on hardware, higher zero-shot success than the HER recipe on at least one multi-stage task.

Why not just…?

Why not just hand-design a dense reward? For lifting and pushing, this may work as well as our method, and we treat that as a real threat. The evidence against it is partial. DrEureka reports that automated rewards beat expert-designed ones, and AIRL's authors note that hand rewards invite reward hacking. Neither shows failure under randomization specifically. Robosuite's staged rewards give us a direct test. If they match $g_\theta$ on Stack and Nut Assembly at held-out physics, the method's value shrinks to cases where no staged reward exists, and we will report that.

Why not just run plain AIRL inside the randomized simulator? This is the obvious idea, and our central ablation tests it. The argument predicts that a potential blind to $\xi$ forces physics-dependent value structure into $g_\theta$. If plain AIRL performs as well, the idea collapses to "run AIRL with randomization." We then report that as a simpler positive finding and drop the privileged potential.

Why not just use HER with a recurrent policy, which already works on hardware? It works on single-stage puck pushing. The file gives no evidence that it solves multi-stage contact tasks: SURREAL reports Nut Assembly unsolved, and the bimanual study needed about a billion steps to reach 50% real success (Bi-Manual Block Assembly via Sim-to-Real Reinforcement Learning). The randomization-width sweep below tests whether HER degrades faster than $g_\theta$ as ranges widen.

Why not just narrow the randomization until sparse RL works? Narrow ranges are what break transfer, as DrEureka's account of the dilemma between unlearnable and overfit ranges describes. The width sweep makes this trade-off measurable for every method.

Proposed Experiments

The first stage is a diagnostic in MuJoCo Ant, randomized over mass, friction, and per-leg actuator strength. Demonstrations come from a state-based expert trained per physics sample. We follow AIRL's protocol: learn the reward, retrain a fresh policy on it under held-out dynamics including the two-leg-disabled Ant, and report return. AIRL's 130.3 on Disabled Ant is quoted for context. Our setting trains over a family rather than one source, so the numbers are not directly comparable. We also measure how well $g_\theta$ ranks states compared with the true task reward under each held-out $\xi$.

The second stage is the main benchmark: robosuite Lift, Stack, Door, and Nut Assembly on Panda under operational-space control, with 5 to 20 teleoperated demonstrations, sparse success reward, and the randomization above. We report success on held-out physics inside and outside the training ranges, plus success against simulator steps.

The third stage is hardware on a Panda arm with object pose tracking. We test zero-shot puck pushing and block stacking, then 200 real fine-tuning episodes rewarded by $g_\theta$. The 200-episode budget matches the budget VLAC reports for going from 30% to 90%.

Three checks are decisive. First, the width sweep: as randomization ranges grow, the transfer quality of $g_\theta$ should rise or hold while the HER baseline's sample efficiency falls. Without this crossing, the identifiability argument fails. Second, the ablation with and without $\xi$ in the potential should open a gap that grows with width. Third, we audit for reward hacking: policies trained on $g_\theta$ alone, without sparse success, must reach real task success rather than just high $g_\theta$.