Robots trained by reinforcement learning usually learn in simulation and are then deployed on hardware with no further training. To make that jump work, practitioners use domain randomization: during training they vary physical parameters such as mass, friction, and motor strength, so the policy cannot overfit to any single simulated world. They also face a second choice. Fast simulators get contact and actuator behavior wrong in some regimes. Accurate simulators get it right but run far too slowly for the tens of millions of steps that training needs. The usual compromise picks a single simulator, or a fixed mixing ratio between two, and applies it uniformly across the whole randomized parameter space.
This proposal argues that the expensive simulator should be spent unevenly across the randomization distribution. It should go to the parameter regions where the fast and accurate simulators disagree about how well the current policy performs, and almost nowhere else. We propose to run fast and accurate rollouts in pairs that share the same random seed and initial state. The paired difference serves as a statistical correction, a control variate: a cheap, correlated quantity that cancels most of the noise in an expensive estimate without biasing it. This correction lets the training objective stay unbiased at the accurate fidelity while most episodes stay cheap. Choosing how many accurate rollouts each parameter region gets then becomes a classical allocation problem with a closed-form answer. The aim is zero-shot transfer: no real-world rollouts during training, and a "reality" that neither simulator models exactly.
Core Claim
The domain-randomized training objective is an average of policy return over a distribution of physical parameters $\xi$. Write $J_h(\xi)$ for the return under the accurate, high-fidelity simulator and $J_\ell(\xi)$ for the fast, low-fidelity one, both for the current policy. We split the parameter space into regions $r$ that carry randomization mass $p_r$. In each region we estimate the high-fidelity average from many cheap rollouts plus a correction from a few paired rollouts.
$$\hat J_h = \sum_r p_r \Big[ \bar J_\ell^{(r)} + \beta_r\big(\bar J_h^{(r)} - \bar J_\ell^{(r),\text{paired}}\big)\Big]$$
Here $\bar J_\ell^{(r)}$ is the mean over all cheap rollouts in region $r$. The term $\bar J_h^{(r)} - \bar J_\ell^{(r),\text{paired}}$ is the mean difference over the paired rollouts in that region. The coefficient $\beta_r$ is chosen per region to minimize variance, as in A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation. Suppose the paired discrepancy in region $r$ has standard deviation $\sigma_r$ and an accurate rollout costs $c_h$. Standard stratified-sampling arguments then give the variance-minimizing number of accurate rollouts for a fixed budget.
$$n_h^{(r)} \propto \frac{p_r\,\sigma_r}{\sqrt{c_h}}$$
Two consequences follow from this rule. First, high-fidelity compute concentrates where randomization mass meets fidelity disagreement, for example regimes of high friction, light links, or hard impacts, where coarse integration misbehaves. Second, the rule degrades gracefully. Where pairing correlation is low, the optimal $\beta_r$ shrinks toward zero and that region falls back to pure accurate rollouts. The method therefore never trusts a bad cheap simulator more than its measured correlation justifies.
The same estimator applies to the policy gradient. We would apply it per region inside REINFORCE, where the MFPG unbiasedness argument holds exactly. Because $\sigma_r$ changes as the policy changes, the allocation is re-estimated every few iterations from a small running set of paired rollouts.
The claim is falsifiable. If the fidelity discrepancy is spread evenly across $\xi$, the rule reduces to uniform allocation and should offer no advantage over a well-tuned fixed mixing ratio. The method earns its keep only when the discrepancy is concentrated. We will test this directly.
Relation to Prior Work
Two existing papers motivate the proposal by showing that where effort goes inside the randomization distribution matters. Active Domain Randomization learns a sampler that steers training toward parameter settings whose trajectories differ from a reference environment. It found that uniform randomization wastes effort on settings the policy has already mastered. Learning Domain Randomization Distributions for Transfer of Locomotion Policies learns the randomization range itself and reports reaching Hopper reward above 3000 in roughly half the epochs of fixed uniform randomization. Both papers change which parameters are sampled, and both assume every rollout costs the same and comes from one engine. We leave the randomization distribution fixed and change which simulator evaluates each sample. The two ideas are complementary, and we test whether they combine.
Three papers supply the multi-fidelity machinery, and each allocates over a different axis than we do. Virtual vs. real: Trading off simulations and physical experiments in reinforcement learning with Bayesian optimization models real return as simulated return plus a learned error term. It trades 1-second simulations against 30-second hardware trials while tuning two LQR gains. It allocates over policy parameters, uses one fixed simulator, and does not scale to neural policies. Sim and Real: Better Together trains on a simulator and a real environment simultaneously using fixed collection and replay rates. The authors name adaptive rates and more than two environments as open problems. Their allocation is over time, with a single global ratio. MFPG introduces the paired-rollout control variate we reuse, but it uses a fixed off-dynamics simulator with no randomization and no cost-driven allocation. Our contribution is to allocate across the randomized parameter space, using the per-region discrepancy that MFPG's estimator already exposes.
Bayesian Domain Randomization for Sim-to-Real Transfer and Data-Efficient Domain Randomization With Bayesian Optimization also optimize randomization using an expensive evaluator, but that evaluator is hardware, with 700 real rollouts across 20 policies. They also retrain the policy at every outer step. Our method needs no outer loop and no hardware during training.
Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation uses simulation as a control variate when evaluating policies, and reports a confidence target reached with 138 rather than 807 real samples. That result is strong evidence that cross-platform correlation is exploitable, but the paper only evaluates and does not train.
One paper constrains the design. MFPG reports that off-dynamics RL methods which consume low-fidelity transitions directly fail badly when those transitions are misspecified. We therefore never train directly on uncorrected cheap returns in regions where their discrepancy is large.
Research Program
We would build a two-tier simulator hierarchy inside MuJoCo, using the RRLS parameter-modification interface (SuReLI/RRLS).
The low tier uses a large integration step, few contact-solver iterations, and ideal torque actuators. The high tier uses a small step, many solver iterations, and a DC-motor model with 15–19 ms latency, following the actuator modeling in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. We measure the cost ratio $c_h/c_\ell$ in wall-clock time rather than assuming it.
Zero-shot "reality" is a third simulator that neither tier sees during training. It has the high-tier physics plus held-out effects: gear backlash, sensor noise, and parameter values just outside the training range.
The tasks are Hopper, Walker2d, HalfCheetah, and Ant, together with a MuJoCo peg-in-hole insertion task, where contact discrepancy should be most concentrated. The parameter regions come from a coarse grid over the two or three randomized parameters with the largest effect. For higher-dimensional randomization, the regions instead come from clustering the paired discrepancies.
The work has three stages. First, we measure how concentrated the fidelity discrepancy is across $\xi$ for fixed policies, before any allocation. This tells us in advance which tasks should benefit. Second, we implement the allocated estimator inside REINFORCE, where its unbiasedness is exact, and measure the variance of the gradient estimate at equal cost. Third, we port it to a PPO surrogate. There the per-region correction applies to advantage-weighted returns, and unbiasedness no longer holds exactly. We report how much bias this introduces by comparing against gradients computed from high-fidelity rollouts alone.
As a stretch hardware check, we would use the Poppy Ergo Jr. pusher from Active Domain Randomization. The high tier would be a contact-refined simulator, and evaluation would use the real arm only.
Baselines
The naive reference is uniform domain randomization trained entirely in the low-fidelity tier, given the same total cost budget. This is what most practitioners do. It shows how much the fidelity gap alone costs at transfer.
The second reference is uniform randomization entirely in the high-fidelity tier at the same budget. It sees the right physics but far fewer episodes.
The strongest competitor is fixed-ratio mixing of the two tiers, in the style of Sim and Real: Better Together, with collection and replay rates as hyperparameters. We give it an oracle advantage: a grid of five high-fidelity fractions, each run with full seeds, reporting the best fraction chosen on the reality simulator. Our method gets no such tuning.
On its own Fetch Push setting, that paper reports reaching a 1.0 success rate with about 10,000 real episodes at a real collection rate of 0.1, versus about 40,000 real-only, while sim-only fails entirely. Those numbers come from a sim-plus-real setting, not our two-simulator one. We therefore re-run the method rather than compare to the figures.
MFPG with uniform pairing across $\xi$ and no allocation is the closest method baseline. On MuJoCo it reports substantial variance reduction under mild dynamics gaps, but no scalar result was found on Lacuna for MFPG's returns. We also include Active Domain Randomization run in the low tier. No reported result for it on our tasks was found on Lacuna.
Two ablations isolate the key ingredient. The first keeps the control-variate correction but allocates accurate rollouts uniformly across regions. The second keeps the discrepancy-driven allocation but drops the correction, simply mixing raw high and low episodes in the allocated proportions.
All methods share the same policy architecture, randomization distribution, cost budget in measured simulator-seconds, ten seeds, and an equal hyperparameter search of at most five configurations for each method. The fixed-ratio grid is the exception, being deliberately generous.
Beating the strongest baseline means one of two outcomes on the held-out reality simulator. Either the method reaches the oracle-tuned fixed ratio's final return using at most half its budget, or it achieves higher final return at equal budget, with non-overlapping 95% bootstrap intervals over seeds. Matching the oracle without tuning would be a weaker but still useful result.
Why not just…?
Why not just widen the randomization in the cheap simulator until it covers the accurate one? This is the most natural fix: treat the fidelity gap as one more source of uncertainty. Learning Domain Randomization Distributions for Transfer of Locomotion Policies shows the cost of over-wide ranges, which include unsolvable settings and doubled training time on Hopper. More fundamentally, coarse integration errors are often structured rather than random. Stiff contacts that are too bouncy, or missing latency, are not reachable by perturbing mass or friction. We test this directly with a baseline that searches for the widest cheap-tier ranges, under the same tuning budget. If it matches our method on the reality simulator, we will report that the fidelity gap on these tasks is absorbable by randomization.
Why not just tune a fixed high/low ratio once? This is the real threat, which is why the fixed-ratio baseline receives oracle tuning. The allocation rule predicts gains only when the discrepancy $\sigma_r$ is concentrated in a few regions. We will therefore construct task variants that sweep concentration, from discrepancy spread evenly across all regions to discrepancy confined to a single region. The advantage over the best fixed ratio should grow with concentration and vanish at the uniform end. If it does not vanish there, something other than allocation is driving the gains. If it never appears, the proposal is wrong.
Why not just pre-train in the cheap simulator and fine-tune in the accurate one? Sequential schedules are standard and cheap to implement. They spend accurate compute at the end of training, uniformly over $\xi$. Sim and Real: Better Together found that sim-only pre-training could leave policies unrecoverable under a large friction gap, which suggests timing matters. We run the sequential schedule with three handover points at equal budget. If one of them matches our method, the lesson is that the time axis is enough and the parameter axis is unnecessary.
Decisive Checks
The first check comes before any training. Paired-rollout correlation between tiers, measured per region and across horizons, must remain well above zero in the high-discrepancy regions. If it collapses because of chaotic divergence over long horizons, the control-variate premise fails. We would then shorten the pairing to truncated segments before proceeding.
The second check is the concentration sweep described above. It is the cleanest test of whether allocating across parameter space is what matters.
The third check is transfer to the held-out reality simulator. It must come out in our favor relative to the oracle-tuned fixed ratio. Lower training variance alone does not count as success.
The fourth check is optional and uses the Poppy Ergo Jr. pusher. We would compare success rates over twenty hardware trials for our method, the best fixed ratio, and low-tier uniform randomization, with no real-world training data used by any of them.
References
- A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation
- Active Domain Randomization
- Learning Domain Randomization Distributions for Transfer of Locomotion Policies
- Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization
- Sim and Real: Better Together
- Data-Efficient Domain Randomization With Bayesian Optimization
- Leveraging Correlation Across Test Platforms for Variance-Reduced Metric Estimation
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots