Domain randomization trains a robot controller in simulation while randomly varying physical constants such as mass, friction, and motor delay, so that the controller works on real hardware it has never touched. It has a known failure: when the ranges are wide, training becomes unstable and slow. Off-policy learners like soft actor-critic (SAC) can fail outright (Cyclic Policy Distillation), and naive randomization reaches 0.3% success on YCB Pick & Place where a structured method reaches 73.8% (Rapid Motor Adaptation for Robotic Manipulator Arms). A separate literature shows how to wring much more learning out of each simulator step. These methods reuse replayed experience many times per step, a high "update-to-data" (UTD) ratio. They stay stable by periodically resetting networks and by running offline "stabilization" phases. None of this has been tested where data comes from many different physics.
We argue these two problems share one cause. Suppose the value function (the critic, which predicts future reward) does not see the randomized physics parameters. Then the same state and action carry conflicting reward targets depending on which world produced them. Replaying those transitions many times amplifies the conflict. Give the critic the parameters, and the replay buffer becomes a consistent dataset over a continuum of physics. It can then be squeezed hard with high-UTD training, and a deployable student that sees only sensors can be distilled from it offline. The goal is a method that trains in fewer simulator steps and less wall-clock time than prior domain-randomization pipelines in simulators too slow for brute-force parallel PPO (proximal policy optimization, the standard on-policy method), such as contact-rich manipulation.
Core Claim
Write $\mu$ for the vector of randomized physics parameters of an episode, $s$ for simulator state, and $a$ for action. A critic $Q(s,a)$ trained on a buffer mixing many $\mu$ regresses toward an average over incompatible worlds. The missing information makes the target non-Markov, and more gradient steps on the same data cannot fix that. A critic $Q(s,a,\mu)$ has well-defined targets. The claim is specific and falsifiable.
First, under wide randomization, the asymptotic shortfall of high-UTD off-policy learning relative to a fixed-physics specialist grows with UTD when the critic lacks $\mu$. With $\mu$ it stays roughly flat.
Second, resets fix plasticity loss only in the $\mu$-conditioned case. In the unconditioned case the bottleneck is target inconsistency, not plasticity.
Third, once the critic is consistent, the most expensive part of prior distillation pipelines can move off-policy. Distilled Domain Randomization spends on-policy rollouts to distill its teachers into a student; here most of that distillation runs on replay during offline phases.
We measure the shortfall with a specialist gap. For held-out parameters $\mu$, let $J_{\text{spec}}(\mu)$ be the return of a policy trained only on $\mu$ and $J(\mu)$ the return of the method under test:
$$\Delta = \mathbb{E}_{\mu \sim p_{\text{test}}}\left[1 - J(\mu)/J_{\text{spec}}(\mu)\right].$$
Part of $\Delta$ is the unavoidable price of acting without knowing $\mu$. The rest is an optimization artifact, and the proposal targets that part.
Relation to Prior Work
The motivating evidence comes from two papers that found domain randomization's instability under wide ranges. Cyclic Policy Distillation splits the range into sub-domains, trains local policies in a fixed cycle, and distills them; its sequential cycle cannot use parallel GPU simulation, and it needs hand-chosen partitions. Distilled Domain Randomization trains specialist teachers on fixed parameter samples and distills them into one student using the student's own rollouts. Both treat the conflict between worlds as something to avoid by splitting worlds apart. We instead remove the conflict inside one parameter-conditioned critic, which acts as a continuous family of teachers trained from a single shared buffer across all parallel environments.
The method draws on three sources. Breaking the Replay Ratio Barrier supplies the reset schedule (SR-SAC at replay ratio 128 reaches IQM 740 on DMC15 versus 391 for SAC). It also supplies the finding that a small stream of fresh online data stabilizes aggressive reuse. SPEQ supplies the alternation of cheap online phases with offline high-UTD phases on a frozen buffer. It reports 27–78% less training time on MuJoCo but leaves the phase schedule hand-tuned. Aggressive Q-Learning with Ensembles supplies a low-cost bias control: multi-head ensembles whose targets average the lowest estimates.
Privileged training is the closest prior art, and it constrains the design. Asymmetric Actor Critic gives the critic full simulator state while the actor sees images. That state does not include $\mu$, and the method was run at low UTD. RMA² conditions a teacher on encoded physics and then trains an adapter from sensor history. It uses low-UTD training, and it reports a remaining gap between adapter and oracle. We use the same teacher–student split. What is new is the regime (high-reuse off-policy learning with resets and offline phases) and where the distillation happens: inside the offline phases, on replay data.
Research Program
The system runs $M$ parallel simulator copies, each with its own sampled $\mu$. Every transition is stored with its $\mu$.
A privileged teacher actor $\pi_T(a \mid s,\mu)$ and a multi-head critic $Q(s,a,\mu)$ train at low UTD during online phases. Every $K$ environment steps, collection pauses for an offline phase. In that phase the critic and teacher run many updates on the frozen buffer. The student $\pi_S(a \mid o_{1:t})$, which sees only a history of observations such as proprioception and depth, is regressed onto the teacher's actions at buffer states. Buffer states come from teacher rollouts, so the student would otherwise drift at deployment. To correct this, a small fraction of online episodes are driven by the student and labeled by the teacher. Networks are reset on the schedule from the replay-ratio paper.
We will replace SPEQ's hand-set schedule with one simple signal: the critic's Bellman error on a held-out slice of the buffer. When that error stops falling, the offline phase ends. We treat this as a secondary contribution and will report fixed schedules alongside it.
The analysis half matters as much as the method. For each benchmark we sweep randomization width, UTD ratio, whether the critic sees $\mu$, and whether resets are on, and we measure $\Delta$ at matched simulator steps and matched wall-clock. This decomposes the asymptotic gap into a part caused by missing $\mu$ in the critic, a part caused by plasticity, and a residual caused by the student's partial observability. The residual is the adapter–oracle gap that RMA² leaves unexplained.
Baselines
The naive reference is uniform domain randomization with the standard learner for each setting: SAC with a single unconditioned critic on MuJoCo, PPO on ManiSkill2 and Isaac Gym. Its reported outcomes are that SAC-DR "failed to learn" wide-range HalfCheetah within practical budgets (Cyclic Policy Distillation) and that standard DR reached 0.3% on YCB Pick & Place (RMA²).
The strongest existing method on manipulation is RMA², a privileged teacher plus a sensor-history adapter. It reports 73.8% success on YCB Pick & Place, 90.5% on EGAD, and 51.6% on 3 mm Peg Insertion, with 4x faster training than end-to-end vision RL. We will re-run it in the same ManiSkill2 task versions, with the same randomization ranges, simulator step budget, and GPU budget. Whether our re-run matches the reported numbers will be reported separately from the comparison.
On locomotion under wide randomization, the strongest baselines are Cyclic Policy Distillation and Distilled Domain Randomization. We will run both with the same number of total simulator steps. Distilled Domain Randomization's on-policy distillation rollouts count against its budget. No reported quantitative success percentage was found on Lacuna for its hardware results.
We also add PPO with a critic that sees $\mu$, trained at full GPU parallelism. It is the obvious modern competitor, and we discuss it below.
The key ablations isolate the proposal's ingredient. The first is the same high-UTD, reset, and offline-phase pipeline with an unconditioned critic. The second is the same pipeline with the critic seeing $\mu$ but no resets. The third is the same pipeline with the student distilled on-policy only.
Every method gets an equal hyperparameter search budget on a validation set of $\mu$ samples that is disjoint from the test set. Beating a baseline means lower $\Delta$ on held-out $\mu$, or equal $\Delta$ reached in less simulator time or wall-clock. All results are averaged over at least five seeds and summarized with interquartile means, as in the replay-ratio work.
Why not just…?
Why not just run PPO with a privileged critic across thousands of GPU environments? This is the real threat. Isaac Gym reports up to 700K steps per second on Ant. Benchmarking Massively Parallelized Multi-Task RL finds PPO often wins in wall-clock at that throughput. We do not expect to beat it there. Our claim is about simulators where steps are expensive. ManiSkill2 reports about 2,532 steps per second with a Nature CNN, and in that setting PPO's appetite for samples dominates wall-clock. The experiments locate the crossover explicitly. We run both methods across simulators spanning roughly 10³ to 10⁵ steps per second and report wall-clock to a fixed success threshold. If PPO with a $\mu$-aware critic wins even at ManiSkill2 throughput, the method claim fails, and only the diagnostic result survives.
Why not just raise the UTD ratio and add resets, without the parameter-conditioned critic? SR-SAC's gains were shown on fixed-physics tasks. Our hypothesis is that under wide randomization the failure is inconsistent targets, which resets cannot repair. The unconditioned high-UTD ablation tests this directly. If it matches the full method, the core claim is wrong. We will then report that plasticity, not the missing information about physics, was the bottleneck.
Why not just use RMA² as is? RMA² already has a privileged teacher. It trains at low reuse, however, and leaves an unexplained gap between adapter and oracle. Our contribution must show up as fewer simulator steps to reach RMA²'s success rates, or as a smaller residual gap. If neither appears on ManiSkill2 Pick & Place and Peg Insertion, the added machinery is not justified.
Why not just deploy an ensemble of specialists? Distilled Domain Randomization shows ensembles break the 2 ms control loop of fast hardware, which is why every method here deploys a single student.
Proposed Experiments
The first decisive check runs on MuJoCo HalfCheetah and Hopper with the wide mass and friction ranges used by Cyclic Policy Distillation. It plots $\Delta$ against UTD in {1, 5, 20, 64} for critics with and without $\mu$. The core claim predicts a widening gap only without $\mu$.
The second check runs on ManiSkill2 YCB Pick & Place and 3 mm Peg Insertion. It compares against RMA², reporting success rate against simulator steps and wall-clock, and it measures the student–teacher gap as a function of the fraction of distillation done offline.
The third check runs Isaac Gym Ant and Humanoid, with simulated throughput throttled to several levels. It maps where our method and PPO with a $\mu$-aware critic cross over in wall-clock to threshold.
A final transfer check uses the ManiSkill2 PickCube sim-to-real setup, where 91% success in simulation fell to 60% zero-shot on hardware (ManiSkill2). It tests whether the lower specialist gap in simulation shrinks that drop, at equal training budget.
References
- Cyclic policy distillation: Sample-efficient sim-to-real reinforcement learning with domain randomization
- Rapid Motor Adaptation for Robotic Manipulator Arms
- Distilled Domain Randomization
- Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier
- SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning
- Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
- Asymmetric Actor Critic for Image-Based Robot Learning
- Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning
- Benchmarking Massively Parallelized Multi-Task Reinforcement Learning for Robotics Tasks
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills