Robots trained in simulation usually fail on real hardware because the simulator's physics and sensors are wrong in small ways. The standard fix is domain randomization: during training, the simulator's friction, mass, damping, and latency are varied so the policy cannot rely on any single setting. In practice the variation is widened gradually as a curriculum, an easy-to-hard training schedule. Every such schedule today is a hand-tuned rule. Adaptive Curriculum Generation from Demonstrations widens randomization when success exceeds 60% and narrows it below 40%. Active Domain Randomization lets a discriminator decide which settings to train on. Nobody can say in advance how wide to go, how strongly each stage should stay close to the previous one, or when further widening only makes the policy sluggish without helping on the real system.
We propose to answer these questions with the replica method. This is a tool from the physics of disordered systems that computes the exact average behavior of a large learning problem whose data are random. It works by analyzing several identical copies ("replicas") of the learner trained on the same random draw. We treat the randomized simulator parameters as the frozen randomness. We model each curriculum stage as a regularized regression that is pulled toward the previous stage's solution. Under these modeling choices, the whole staged curriculum becomes a chain of transfer-learning problems that the replica method can solve in closed form. The output is a phase diagram for real-system error as a function of randomization width, number of stages, anchoring strength, data per stage, and the structure of the sim-to-real mismatch. That diagram yields an explicit pacing schedule and a stopping point, which we then test on nonlinear control benchmarks.
Core Claim
Real-system error under staged randomization splits into three parts. The first is a part of the sim-to-real mismatch that randomization can reach, because it lies along directions the randomized parameters actually move. The second is a part it cannot reach, which is physics missing from the simulator. The third is a variance term that grows with randomization width and shrinks with data and with anchoring to the previous stage.
The replica calculation makes this decomposition quantitative, and it predicts three testable things. It predicts an optimal final width that scales with the size of the reachable mismatch, not with the total mismatch. It predicts an optimal per-stage anchoring strength that depends on noise level, in the spirit of the "either/or" rule found for one-step transfer. It predicts a sharp threshold beyond which widening raises real error. That threshold is the formal version of the "over-conservatism of wide randomization" that practitioners report.
The new element is not a better randomization heuristic. It is an analytical prediction of where the widening curriculum should stop and how hard each stage should hold onto the last. We check that prediction against grid search on nonlinear systems.
Relation to Prior Work
Three lines of work motivate the gap.
- Adaptive randomization curricula with no theory. ACGD, ADR and SS-ADR show that these curricula beat uniform randomization on real arms. None of them predicts the right width or pacing.
- Theory with no curriculum. Domain Randomization is Sample Efficient for Linear Quadratic Control proves that randomization matches certainty equivalence's $1/N$ excess-cost rate for linear quadratic control while avoiding its low-data instability. It handles a single fixed randomization distribution, and its nonlinear pendulum result is empirical only.
- The method we reuse. Transfer Learning in ℓ1 Regularized Regression uses the replica method to compute exact generalization error for one-step transfer from auxiliary data. Its predictions match simulations at 16,000 features. It shows that standard Trans-Lasso can be about 30% worse than tuned alternatives. It also shows that a simple either/or rule lands within about 10% of the optimum.
We extend that analysis in three ways:
- from one transfer step to a chain of $K$ stages;
- from a fixed auxiliary dataset to a source whose randomness is itself controlled by the curriculum;
- from i.i.d. regression targets to targets generated by a nonlinear dynamical system.
Two strands supply the technical machinery. The replica analyses of Lasso resampling and bootstrap averages treat random sample weights as frozen disorder. We treat random simulator parameters the same way. Learning curves for deep structured Gaussian feature models supplies the random-feature learning curves we use to represent nonlinear targets.
Two results constrain the design. First, replica derivations assume Gaussian-like data, which closed-loop robot data violate. The theory is therefore stated for learning from a fixed state distribution, as in behavior cloning or model learning, and closed-loop validity is tested empirically rather than claimed. Second, non-convex learners break the "all copies behave alike" assumption, as in Discontinuous Generalization in Large Committee Machines and Replica Symmetry Breaking in Compressive Sensing. We therefore restrict the exact theory to convex stage problems and report only where that assumption provably fails.
Research Program
The solvable model
A learner fits a controller or one-step dynamics model $\hat y = w^\top \phi(x)$, where:
- $\phi$ is a fixed nonlinear random-feature map of state and action;
- labels come from a nonlinear target $g(x;\theta)$, such as an expert controller or the true next state;
- the simulator parameter $\theta$ is drawn as $\theta \sim \mathcal N(\theta_0, \sigma_k^2 I)$ at stage $k$.
Stage $k$ solves
$$w_k = \arg\min_w \frac{1}{n_k}\sum_{i=1}^{n_k}\big(g(x_i;\theta_i) - w^\top\phi(x_i)\big)^2 + \lambda_k\|w - w_{k-1}\|^2 + \gamma\|w\|^2,$$
where $n_k$ is the number of samples at stage $k$, $\lambda_k$ is the anchoring strength toward the previous stage, and $\gamma$ is ordinary ridge regularization. The real system has parameter $\theta^\star$ together with an unmodeled residual $r(x)$. The quantity we predict is the real error $E_{\text{real}}(w_K)$ on held-out real states.
Expanding $g$ around $\theta_0$ splits the real-minus-nominal target into two pieces. The reachable piece is the projection of $g(x;\theta^\star)-g(x;\theta_0)$ onto the span of $\partial_\theta g$. The unreachable piece is everything else, including $r$. The replica computation, under Gaussian equivalence of the features, tracks a small set of summary quantities across stages:
- $m_k$, the alignment of $w_k$ with the real target;
- $q_k$, the agreement between two replicas trained on independent randomization draws;
- $c_k$, the overlap between successive stages $w_k$ and $w_{k-1}$.
Real error is then a closed-form function of $(m_K, q_K)$. This turns the choice of schedule $\{\sigma_k, \lambda_k, n_k\}$ into a low-dimensional deterministic optimization.
From theory to schedule
Solving that optimization gives a predicted optimal width path, a predicted anchoring sequence, and a predicted stopping width $\sigma^\dagger$. Past $\sigma^\dagger$, the variance added by wider randomization outweighs the reachable bias it removes.
The theory also identifies a measurable signal. The replica agreement $q_k$ is the agreement between two learners trained on independent randomization draws at the same width, and it can be estimated cheaply during training. Under the theory, its trajectory together with $c_k$ indicates when the next widening step stops paying. This gives an online pacing rule calibrated by the theory, not by a tuned threshold.
For the non-convex case, we compute the point where the replica-symmetric solution becomes unstable. We test whether real learners with small two-layer policies show a sharp jump in replica disagreement there.
Nonlinear systems
The theory is checked on progressively less idealized targets:
- a synthetic high-dimensional teacher at the 16,000-feature scale used in the Trans-Lasso replica study;
- the 3D linear system and the receding-horizon inverted pendulum of the LQR domain randomization paper;
- closed-loop policies on LunarLander-v2 (randomized engine strength) and Pusher-3DOF-v0 (randomized friction and damping), as used by ADR;
- the ErgoReacher simulator with its 8-dimensional parameter space.
"Reality" in all of these is a held-out simulator with shifted parameters plus deliberately unmodeled effects: actuation latency, backlash, and stick-slip friction. This lets us control the reachable and unreachable mismatch separately. Physical Poppy Ergo Jr. transfer is an optional final step if hardware is available.
Baselines
Naive references. The do-nothing reference trains on the nominal simulator $\theta_0$ with no randomization. Its real error measures the full gap. The second reference is uniform domain randomization at a single wide range chosen in advance. This is the default practice, and the theory predicts it overshoots $\sigma^\dagger$. On the linear system, certainty equivalence and robust control serve as references, because the LQR paper characterizes both: certainty equivalence suffers burn-in instability, and robust control remains conservative even with infinite data.
ACGD-style success-threshold curriculum. We re-implement this curriculum with its 60%/40% widening and narrowing rule on each benchmark. It is the most direct competitor because it is the simplest adaptive pacing rule with real-robot evidence. It reports 85% real success on KUKA pick-and-stow and 60% on block stacking (ACGD). We cannot run that exact KUKA setup, so those numbers are context, not a target.
Active Domain Randomization. ADR is the strongest existing method on our shared benchmarks (LunarLander-v2, Pusher-3DOF-v0, ErgoReacher). The file reports only qualitative results for it; no reported numerical result was found on Lacuna for ADR on these tasks. We therefore rerun ADR ourselves.
Fair comparison. All methods use the same simulator, the same total number of simulator samples, the same policy architecture, and the same hyperparameter tuning budget, measured in training runs. Each method gets its own budget for its thresholds or particle counts. Real-proxy evaluation is never used for tuning.
Ablations. Three ablations isolate the proposal's ingredients:
- the theory-predicted width schedule with anchoring set to zero, which is plain staged warm-starting;
- theory-predicted anchoring with a hand-picked linear width ramp;
- the online pacing signal replaced by generic ensemble disagreement with a tuned threshold.
What counts as beating a baseline. We claim success if the theory-derived schedule matches or beats the best baseline on real-proxy return or error across seeds, with no tuning against the real proxy. A schedule tuned by grid search on the real proxy is the oracle we aim to approach.
Why not just…?
Why not just use uniform randomization over a wide range? Prior evidence says it is unstable in high dimensions. ADR reports high-variance, forgetting-prone uniform randomization on the 8-dimensional ErgoReacher and complete failure in low-engine-strength LunarLander. Our theory predicts something sharper: wide uniform randomization is worse in exact proportion to how far its width exceeds $\sigma^\dagger$. The width sweep tests this directly by plotting real error against width and comparing the measured optimum to the predicted one.
Why not just tune the width by grid search? Grid search needs repeated evaluation on the real system, which is exactly the scarce resource. It also says nothing about anchoring or pacing. We include grid search on the real proxy as an oracle. If the theory's predicted width lands within the grid's noise band without touching real data, the theory pays for itself. If it does not, the theory has failed its central test.
Why not just widen when an ensemble of policies starts to disagree? This is the real threat. Our online signal is a specific form of ensemble disagreement, and a well-tuned generic version might pace as well. The ablation that swaps in tuned generic disagreement settles this.
Even if generic disagreement matches on pacing, it supplies neither the anchoring strength nor the predicted stopping width. Those remain the claims that distinguish this proposal. We would then present the work as a predictive theory of anchoring and stopping, not as a better pacing signal.
Why not just identify the real parameters and narrow the randomization? Posterior approaches such as Neural Posterior Domain Randomization shrink the reachable mismatch using real data. They cannot shrink the unreachable part: physics absent from the simulator stays absent. Our decomposition predicts the remaining error after identification, so it complements identification rather than competing with it. The experiments vary the unreachable residual to show how real error behaves as that residual grows.
Proposed Experiments
Validating the exact theory. On the synthetic teacher at roughly 16,000 features, simulated real error must match the replica prediction across:
- widths;
- stage counts from 1 to 8;
- anchoring strengths;
- ratios of reachable to unreachable mismatch.
The required accuracy is comparable to the near-perfect agreement reported for one-step transfer (Trans-Lasso replica study). We also test whether the either/or structure reappears per stage: weak anchoring under low noise and strong anchoring under high noise.
Transfer to nonlinear systems. On the pendulum, LunarLander-v2, Pusher-3DOF-v0 and ErgoReacher, we compare two optima:
- the predicted stopping width $\sigma^\dagger$, computed from features fitted on simulator data alone;
- the empirically optimal width found by the real-proxy grid.
Separately, we inject unmodeled latency and backlash. The theory predicts that their effect is a floor on real error that no amount of widening removes, plus a downward shift in $\sigma^\dagger$.
Decisive checks. The proposal fails in any of these three cases:
- the predicted $\sigma^\dagger$ misses the empirical optimum by more than the seed-to-seed spread on two or more nonlinear benchmarks;
- the theory-derived schedule loses to ADR or the ACGD-style rule under equal budgets;
- tuned generic ensemble disagreement matches it on pacing, anchoring, and final error together.
In the non-convex check, if two-layer policies do not show a sharp rise in replica disagreement near the computed instability point, we drop the shattering claim and keep only the convex-stage results.
References
- Adaptive Curriculum Generation from Demonstrations for Sim-to-Real Visuomotor Control
- Active Domain Randomization
- Generating Automatic Curricula via Self-Supervised Active Domain Randomization
- Domain Randomization is Sample Efficient for Linear Quadratic Control
- Transfer Learning in ℓ1 Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis
- Semi-Analytic Resampling in Lasso
- A Statistical Mechanics Approach to Approximate Analytical Bootstrap Averages
- Learning curves for deep structured Gaussian feature models
- Discontinuous Generalization in Large Committee Machines
- Replica Symmetry Breaking in Compressive Sensing
- Neural Posterior Domain Randomization