Robot policies trained in simulation often fail on real hardware because the simulator's physics are wrong: friction, mass, and motor delay all differ from reality. The standard fix is domain randomization, which means training across many randomly perturbed simulators so the policy cannot depend on any single one. Choosing which parameters to perturb, and by how much, decides whether this works. Ranges that are too narrow leave the policy brittle. Ranges that are too wide produce impossible worlds that wash out learning. The automatic methods that pick these ranges all assume the policy is trained by reinforcement learning (RL) against a reward. More and more sim-to-real policies are instead trained by imitation, which copies the actions of a privileged expert: a controller that can see the true simulator state and parameters. Imitation has its own failure mode. Small copying errors push the robot into states the expert never visited, and errors compound from there. Nobody has asked how the randomization distribution should be shaped when the learner is an imitator.
We propose to learn the randomization distribution from two quantities that imitation already provides for free in simulation. The first is whether the privileged expert can still solve the task under a given setting of the physics. The second is how far the student's actions drift from the expert's on the states the student itself visits under that setting. Randomization should concentrate on physics where the expert succeeds but the student diverges. That region is where corrective labels are both valid and needed. Everything outside it is either already learned or not learnable at all.
Core Claim
Let $p_\phi(\xi)$ be a learned distribution over simulator parameters $\xi$, such as friction, masses, damping, and actuation delay, with learnable weights $\phi$. For each $\xi$, let $f(\xi)$ be the privileged expert's success probability. Let $g(\xi)$ be the expected per-step action disagreement $\lVert \pi_s(s) - \pi_e(s;\xi)\rVert$ between student $\pi_s$ and expert $\pi_e$, averaged over states $s$ drawn from the student's own rollouts under $\xi$. We update the distribution by
$$ \max_\phi \; \mathbb{E}_{\xi \sim p_\phi}\big[f(\xi)\, g(\xi)\big] + \alpha\, \mathcal{H}(p_\phi) - \beta\, \mathrm{KL}\big(p_\phi \,\|\, p_{\phi_{\text{old}}}\big), $$
where $\mathcal{H}$ is entropy, which keeps the distribution broad, and the KL term, with weights $\alpha$ and $\beta$, keeps each update close to the previous distribution. We then relabel the student's visited states with expert actions, in the style of DAgger (Ross et al.).
The product $f \cdot g$ has a specific origin. The DAgger analysis bounds the student's excess cost by roughly the horizon times the on-policy disagreement, and that bound only means something where the expert's own cost is low. So $g$ measures remaining imitation error, and $f$ keeps us from spending expert labels on physics where the labels are themselves failures.
The claim is testable. At a fixed budget of simulator steps and expert queries, this distribution should yield imitation students with wider parameter coverage and higher zero-shot hardware success than two alternatives: uniform randomization, and reward-driven automatic randomization transplanted to imitation.
Relation to Prior Work
Two lines of automatic randomization motivate and supply the machinery.
Active Domain Randomization trains a set of samplers, each a particle updated by Stein variational policy gradient. The samplers seek parameters where a discriminator can tell the agent's trajectories apart from those in a reference environment, and the agent is trained with RL. GoFlow represents $p(\xi)$ with a normalizing flow, a density model built from invertible transforms. It trades policy reward against entropy and a KL stability term, and it reached 90% real gear insertion against 60% for full randomization. We reuse GoFlow's flow parameterization and its entropy and KL regularizers essentially unchanged. What we replace is the signal: both methods score parameters by the RL agent's reward or by behavioral novelty. Neither separates "hard for the learner" from "impossible for anyone," and neither uses the dense per-state supervision that imitation offers.
Three papers attack compounding error without solving configuration, and they constrain our design. Action Chunking and Data Augmentation shows that injecting noise into expert demonstrations, and executing actions in open-loop chunks, rivals DAgger. It works in fixed simulators, uses isotropic noise, and names noise shaping as open. Our method can be read as shaping the perturbations: we perturb the physics along directions where the student actually drifts, rather than perturbing states isotropically. Robust Tube MPC imitation gets robustness from one demonstration by labeling states on the boundary of an analytically computed tube, a guaranteed region around the planned trajectory. That requires dynamics models that are linearizable, with bounded disturbances, which contact-rich manipulation lacks. NavOL runs online imitation against a privileged planner in IsaacLab and transfers zero-shot, but it never randomizes dynamics.
Finally, Distilled Domain Randomization trains one RL teacher per sampled domain and distills them into a single student. The domains still come from a hand-set distribution, and the per-domain RL is costly. We use a single privileged expert and let the distribution itself be learned.
Research Program
The expert is any controller with access to the true state and to $\xi$. For robomimic Lift, Can, and Square, it is a scripted or sampling-based controller that is given the true parameters. For the IsaacLab tasks GoFlow uses (Cartpole, Quadruped, and Franka gear insertion), it is a single RL teacher that receives $\xi$ as input. That teacher's training cost is charged to every method that uses it.
We estimate $f(\xi)$ from expert rollouts and fit a small classifier on the results so it can be queried cheaply. We estimate $g(\xi)$ from batched student rollouts. The flow is trained with GoFlow's importance-weighted updates, starting from a deliberately over-wide base envelope. Starting too wide is the failure mode GoFlow itself warns about, and our feasibility term is meant to handle it.
The student is a chunked transformer or recurrent policy of the BC-RNN kind recommended by What Matters in Learning from Offline Human Demonstrations. We also run a secondary study of chunk length $k$, because under shifted dynamics open-loop chunks may stop stabilizing the robot and start amplifying drift. The action-chunking paper leaves that boundary uncharted.
Baselines
The naive reference point is behavior cloning on expert data from the nominal simulator, with no randomization and no relabeling. GoFlow's 30% real no-randomization figure gives a sense of the floor on gear insertion.
The strongest practical competitor is uniform randomization over the same wide envelope, combined with simulator DAgger against the same privileged expert. This is the NavOL recipe with dynamics randomization added, and it is what a practitioner would build today. It receives identical simulator steps, identical expert queries, and the same student architecture, with its envelope width swept under the same tuning budget we spend on $\alpha$ and $\beta$.
The strongest existing configuration method is GoFlow. We run it in two forms. The first is its original RL form, which is the reported hardware reference of 90% gear-insertion success from GoFlow. The second transplants it to imitation by replacing the reward with student task success, while keeping the flow, regularizers, and budget identical to ours. Beating it means matching or exceeding its real success with an imitation student, at equal or lower total simulator steps including expert training, and with higher simulated coverage, meaning the fraction of the parameter box where the student succeeds.
We also run DiDoR-style distillation, with teachers on $N$ fixed domains, on Cartpole, where its hardware results exist. Its RL cost per domain is counted.
Two ablations isolate the key ingredient. The feasibility-only variant samples uniformly from the region where the expert succeeds, ignoring $g$. The gap-only variant drops $f$. For robomimic, no reported result was found on Lacuna for any method under physics randomization. The nominal reference is BC-RNN's 84% on Square from What Matters.
Why not just…?
Why not just use uniform randomization with simulator DAgger? This is the most serious threat, because relabeling alone already removes much compounding error. Our argument is that under wide uniform ranges, a large share of expert labels come from settings where the expert itself fails. Those labels teach the student failures, and the extreme-randomization collapse in CausalWorld suggests such settings dominate as dimensionality grows. The head-to-head comparison at matched query budget settles it. We will report it as a loss if uniform randomization with tuned width matches us.
Why not just inject isotropic noise into demonstrations on the nominal simulator? Noise injection covers state perturbations. A friction or delay mismatch, however, changes which action is correct, not merely which state the robot is in, so no amount of nominal-simulator noise yields labels that are correct under shifted physics. The authors of Action Chunking and Data Augmentation themselves call isotropic noise inefficient in high dimensions. We include this baseline, with matched label count, because it may be competitive on the tasks with the least shift, such as Lift.
Why not just run GoFlow with the student's success as the reward? Success is sparse and binary per episode. It also cannot distinguish a setting the student has not yet learned from one no controller can solve, so the flow either avoids hard but feasible regions or wastes effort on infeasible ones. Still, a reviewer could reasonably expect entropy regularization to compensate. The transplanted-GoFlow baseline and the feasibility-only ablation together test this directly.
Proposed Experiments and Decisive Checks
In simulation, we report coverage and success on robomimic Lift, Can, and Square with 6 to 12 randomized physics parameters, on CausalWorld pushing and picking with every physics variable randomized, and on IsaacLab Cartpole, Quadruped, and gear insertion. Each result is plotted against simulator steps and expert queries.
On hardware, we reproduce the Franka gear insertion setup and run 50 trials per method. Because point estimates from 50 trials are unreliable, we report 95% lower confidence bounds using stochastic_verification, following How Generalizable is My Behavior Cloning Policy?.
The first decisive check is whether the full objective beats the feasibility-only ablation. If it does not, the contribution reduces to "filter randomization by expert feasibility," and we would report that simpler finding as the result. The second check is whether $g(\xi)$ measured in simulation predicts real failure better than offline validation loss does, since the What Matters study found validation loss uninformative. If it does, $g$ doubles as a cheap pre-deployment selection metric. The third check is the chunk-length sweep under the learned versus uniform distributions, which tests whether shaped randomization moves the point at which longer chunks begin to hurt.
References
- Learning monocular reactive UAV control in cluttered natural environments
- Active Domain Randomization
- Flow-based Domain Randomization for Learning and Sequencing Robotic Skills
- Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces
- Demonstration-Efficient Guided Policy Search via Imitation of Robust Tube MPC
- NavOL: Navigation Policy with Online Imitation Learning
- Distilled Domain Randomization
- What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning
- How Generalizable is My Behavior Cloning Policy? A Statistical Approach to Trustworthy Performance Evaluation