How Many Simulated Worlds Are Enough? A Replica Theory of Domain Randomization

Robots trained in simulation often fail on real hardware because the simulator gets small details wrong, such as friction, masses, latency, or rendering. The standard fix is domain randomization: train the policy across many randomly perturbed copies of the simulator so it cannot rely on any one copy's quirks. Practitioners face two choices that are currently settled by trial and error. The first is how wide to make the randomization. The second is how many distinct randomized simulators to sample, as opposed to how much data to collect inside each one. Empirical work shows the cost of guessing wrong. Fixed randomization reaches 0% real-world success on a Franka pushing task where an adaptive method reaches 60% (Domain Randomization via Entropy Maximization). Benchmarks also find that plain randomization is a strong average-case method that still trades away nominal performance (RRLS : Robust Reinforcement Learning Suite). No existing theory predicts where these failures begin.

We propose an exactly solvable model of domain randomization built with the replica method. The replica method is a tool from statistical physics that computes the typical test error of a learner when the number of parameters and the number of samples both grow large at a fixed ratio. Our model has two layers of randomness: a finite set of $K$ sampled simulator instances, and the training data drawn inside each instance. The main new prediction concerns the count of distinct simulators, and it is something no amount of tuning a regularizer can reproduce. There should be a sharp threshold in $K$, set by the number of independent directions the randomization perturbs. Below the threshold, the policy can still find a "shortcut" that works identically on every sampled simulator yet fails on the real system. Above it, the shortcut disappears. The optimal randomization width also shrinks as more of the real-world mismatch falls outside the family of parameters being randomized.

Core Claim

Domain randomization does two different things, and current practice blurs them. When there are infinitely many simulators, randomization acts like a smooth penalty on the parts of the policy that depend on simulator artifacts. When there are finitely many, it acts like a set of $K$ linear constraints that the policy can satisfy while still exploiting artifacts. The question is whether those constraints span the artifact space. We claim three things. First, the transfer error to an unseen "real" domain has a phase-transition-like knee at a critical loading $K/p$, where $p$ is the number of randomized parameter directions. Second, at a fixed total sample budget, spending samples on more distinct simulators beats spending them on more data per simulator until the threshold is reached, and gives diminishing returns after it. Third, the "Goldilocks" optimal width depends on a single misspecification ratio: how much of the real gap lies inside versus outside the randomized family.

The closest existing work is the replica analysis of transfer between two correlated datasets in Probing transfer learning with a model of synthetic correlated datasets. That work studies one fixed source and one fixed target with frozen features. It has no ensemble of sources, no randomization width, and no source count. Statistical Guarantees for Offline Domain Randomization is the only theory of domain randomization in the file. It proves consistency and worst-case gap bounds for estimating simulator parameters from real logs, but it treats policy learning as a black box and gives no learning curves.

The Model

A state $s \in \mathbb{R}^{d_s}$ produces an observation $x = (s,\, G_\xi s)$. The second block holds "artifact features" whose relation to the state depends on simulator parameters $\xi$. Examples are rendering cues, or simulator-specific correlations among sensor channels. The correct action is $y = w^{*} \cdot s + \text{noise}$, and the student policy is linear (or random-features) with weights $w = (w_s, w_a)$.

Domain $k$ has artifact map

$$G_{\xi_k} = G_0 + \sigma \sum_{j=1}^{p} \xi_{kj} E_j + \lambda\, \sigma\, \Delta_k ,$$

where:

  • $G_0$ is the default simulator's artifact map.
  • $E_1,\dots,E_p$ are the $p$ randomized parameter directions.
  • $\xi_{kj}$ are independent standard Gaussians.
  • $\sigma$ is the randomization width.
  • $\Delta_k$ is a random perturbation that leaks into the signal coordinates with strength $\lambda$. It models randomization that also corrupts task-relevant information, which is the reason too-wide randomization hurts.

The "real" domain is

$$G_{\text{real}} = G_0 + \sum_j r_j E_j + \rho R ,$$

where:

  • the $r_j$ are the components of the real gap that lie inside the randomized family;
  • $R$ is the unmodeled component, outside that family;
  • $\rho$ is its size.

The student minimizes ridge or logistic loss over $n$ samples from each of the $K$ domains. The proportional limit sends $d_s$, $p$, $K$ and $n$ to infinity together, with $\alpha = Kn/d$ and $\kappa = K/p$ held fixed.

The replica calculation averages over both levels of disorder: the domains $\xi_k$ and the samples within each domain. Under the replica-symmetric assumption it reduces to a few scalar order parameters. These are the alignment of $w_s$ with $w^{*}$, the weight $w_a$ places inside the span of the $E_j$, the weight it places outside that span, and the self-overlap. Real-domain error is a closed-form function of these quantities. Because the loss is convex, replica symmetry is expected to hold. Gordon-type comparison arguments, as used in Learning curves of generic features maps for realistic datasets with a teacher-student model, could make the ridge case rigorous. The same paper's Gaussian-equivalence result is what lets us apply the theory to random-features policies on correlated inputs. It also limits us: that paper shows Gaussian equivalence fails for nonlinear classification on raw images, so we restrict claims to regression-style action targets on low-dimensional state features.

The mechanism behind the $K$ threshold is simple to state. An artifact weight $w_a$ that is orthogonal to every sampled perturbation $\sum_j \xi_{kj}E_j s$ is invisible to the training loss. Yet it can be correlated with the real gap. The dimension of this "invisible but harmful" subspace shrinks as $K$ approaches $p$. The replica solution tells us whether the shrinkage produces a smooth decline or a sharp knee, and how noise and regularization move it. This is analogous to the specialization transition in Discontinuous Generalization in Large Committee Machines, where error stays flat until sample count scales with the right capacity measure. The analogy is suggestive, not assumed; whether the transition is continuous or discontinuous is an outcome of the calculation.

Research Program

The theoretical work derives the saddle-point equations for ridge and logistic students. It then produces phase diagrams of real-domain error over three planes: width against $\kappa$, misspecification ratio $\rho/\|r\|$ against optimal width, and the split of the total budget between $K$ and $n$. Finally it extracts closed-form approximations for the optimal width $\sigma^{*}$ and the critical loading $\kappa_c$.

The empirical work checks whether these predictions, stated as shapes and scalings rather than exact numbers, carry over to policies trained with reinforcement learning in physics simulators. In those simulators no one controls the artifact structure.

The role of each citation is as follows. Probing transfer learning with a model of synthetic correlated datasets and Transfer Learning in ℓ1 Regularized Regression supply the method. The latter shows that replica learning curves can yield simple near-optimal hyperparameter rules: its "either/or" transfer strategy lands within 10% of the global optimum, while the standard setting is up to 30% worse. We aim for a comparable rule for the width and the simulator count. Domain Randomization via Entropy Maximization and Active Domain Randomization motivate the work, because both document width failures and the difficulty of scaling to many randomized parameters. When resampling/reweighting improves feature learning in imbalanced classification? constrains it. That paper is a warning that intuitive data interventions can have exactly zero first-order effect in these models. Our derivation must therefore check whether small randomization widths help at all, rather than assume they do.

Baselines

The naive reference point is training on the single default simulator, which is fixed or zero-width randomization. On real hardware this reaches 0% zero-shot success on PandaPush (Domain Randomization via Entropy Maximization). In our sim-to-sim tests it sets the floor.

The strongest practical competitor for choosing the width is uniform randomization with the width chosen by grid search on held-out perturbed simulators. It uses the same total environment steps and the same number of seeds as every other method, and the same tuning budget, counted in full training runs, that the theory-derived rule is allowed. The theory-derived rule beats it if it matches the grid-search optimum's held-out return to within seed noise while using a small fraction of the tuning runs. Beyond that, if it correctly predicts which widths fail, we can drop the grid search.

The strongest existing adaptive method is DORAEMON, which maximizes the entropy of the randomization distribution subject to a success-rate constraint. Its strongest reported number in the file is 60% zero-shot success on the physical PandaPush task, against LSDR at 46.7% and AutoDR at 26.7% (Domain Randomization via Entropy Maximization). For the MuJoCo robustness suite, no reported numerical result was found on Lacuna for any method in RRLS : Robust Reinforcement Learning Suite. We will therefore rerun DR, M2TD3 and DORAEMON ourselves under the RRLS protocol of 10 seeds. DORAEMON varies width but never the count of distinct simulators. Beating it means the following: at equal total steps, a fixed-width uniform randomization that uses the theory's recommended $K$ and width reaches equal or higher held-out return, especially on held-out parameters outside the randomized family.

The key ablation isolates the finite-$K$ ingredient. It replaces the $K$ sampled simulators with infinitely fresh randomization whose second moments match, which is equivalent to Gaussian noise on the artifact channels. It also includes an explicitly tuned ridge penalty on the artifact weights. If these match the finite-$K$ results everywhere, the threshold prediction is wrong.

Why not just…?

Why not just let DORAEMON or Active Domain Randomization tune the width automatically? These methods adapt the width online and do well, but they never ask how many distinct simulators are sampled. They also need a hand-set success threshold, and they suffer forgetting when the distribution contracts (Domain Randomization via Entropy Maximization). Our claim is complementary. If the $K$ threshold exists, an adaptive method run with too few distinct simulators will plateau no matter how its width schedule moves. The experiment that tests this runs DORAEMON with a fixed pool of $K$ sampled parameter vectors on either side of the predicted $\kappa_c$. If its held-out return does not show the predicted knee, the theory adds nothing to it.

Why not just treat randomization as regularization and tune a ridge or input-noise penalty? This is the real threat. In the infinite-$K$ limit our model provably reduces to a structured penalty on artifact weights. The proposal is only worthwhile if finite $K$ behaves differently. The synthetic experiments compare finite-$K$ randomization against moment-matched noise and against tuned ridge at the same $\alpha$. The MuJoCo experiments do the same with observation noise matched to the randomized parameters' effect. If matched noise closes the gap at every $\kappa$, we will report that domain randomization in this regime is "just regularization." That would be a clean negative result, but it would not support our design advice.

Why not just sweep width and simulator count empirically? A sweep over two dimensions per task is affordable for one low-dimensional task. It becomes expensive as the number of randomized parameters grows, which is exactly the regime where Active Domain Randomization reports uniform sampling becoming unstable. The theory's value is in predicting how $\kappa_c$ and $\sigma^{*}$ scale with $p$ and with misspecification, so that a sweep at small $p$ predicts the optimum at large $p$. We test this directly: we fit the rule on the RRLS one- and two-parameter uncertainty sets and predict the three-parameter and extended sets.

Why not just do better system identification? Precise identification helps, but Sim-to-Real: Learning Agile Locomotion For Quadruped Robots still needed randomization on top of detailed actuator and latency models. (CAD)$^2$RL found that diverse low-fidelity rendering beat chasing realism. In our model, identification shrinks the in-family gap $r$ but cannot remove the unmodeled component $\rho$. The theory predicts how the optimal width should shrink as identification improves, so the two are complements, not rivals.

Proposed Experiments

The first experiment validates the theory on synthetic data, with $d$ in the low thousands. We compare the replica curves for ridge and logistic students against simulation across width, $\kappa$, $\alpha$ and the misspecification ratio. We report whether the $K$ knee is sharp, where it sits relative to $\kappa = 1$, and whether finite-$K$ and moment-matched-noise curves separate. Agreement in the manner of Learning curves of generic features maps for realistic datasets with a teacher-student model is the minimum bar.

The second experiment is a near-theory bridge. We use behavior cloning of expert actions into random-features policies on the RRLS MuJoCo tasks Hopper, HalfCheetah, Walker2d and Ant, with dynamics parameters randomized over a fixed pool of $K$ instances. This setting is supervised, so the theory applies almost directly. The held-out "real" domains are drawn in two ways: inside the randomized family, and with an additional unrandomized parameter changed, such as damping when only mass and friction were randomized.

The third experiment, which is the decisive one, tests transfer to full reinforcement learning. We train PPO or TD3 with uniform randomization over fixed pools of $K$ simulators at equal total steps and 10 seeds. We vary $K$ around the theory's $\kappa_c$ for $p \in \{1,2,3\}$ from the RRLS uncertainty sets and for larger hand-built sets of up to roughly 15 parameters. Metrics are average and worst-case return on held-out parameters, plus nominal return. The prediction succeeds if three things hold. First, held-out return shows a knee at a $K$ that grows linearly with $p$. Second, past the knee, adding domains helps less than adding steps per domain. Third, the optimal width shrinks when the held-out gap is moved outside the randomized family. It fails if return improves smoothly in $K$ with no $p$-dependent knee, or if moment-matched observation noise reproduces every curve.

If the third experiment succeeds, a final optional check would rerun the PandaPush comparison with DORAEMON's 17 randomized parameters. We would use DORAEMON with a theory-chosen pool size, measured against its reported 60% zero-shot success, though this depends on hardware access we cannot guarantee.