Robot policies are now mostly judged in simulation, because real trials are slow, costly, and sometimes dangerous. A simulator, however, is only a model of the world, and the gap between simulated and real physics is largest when objects are touched, pushed, or rolled. The field currently handles this in one of two ways. Some papers report simulated scores and point to a correlation with real results, such as REALM's Pearson $r = 0.921$. Others run a small batch of real trials, often 10 or 20, and report the mean. Neither approach tells a practitioner how many real trials are actually needed to know a policy's real success rate to within, say, five points, or how much the simulator reduces that number.
This note proposes treating the simulator as a correlated helper measurement instead of a replacement for reality. For every real trial we also run the simulator from the same starting conditions. The simulator's prediction then cancels part of the trial-to-trial noise in the real outcome. The method is a control variate, a standard variance-reduction trick that subtracts a correlated quantity whose mean is known precisely. We wrap it in confidence intervals that hold at small sample sizes and stay valid however biased the simulator is. The result is a concrete accounting of how many real trials one needs. That count shrinks roughly by a factor $1-\rho^2$, where $\rho$ is the per-trial correlation between simulated and real outcomes. We also measure $\rho$ directly, showing where simulators truly save real trials (free-space motion) and where they barely help (contact-rich manipulation).
Core Claim
Let $Y_i$ be the real outcome of trial $i$, such as task progress or tracking error, and let $S_i$ be the simulator's outcome from the matched initial state. Let $\mu_S$ be the simulator's mean, which can be estimated from $M$ cheap extra simulated rollouts. The estimator is
$$\hat\mu = \frac{1}{n}\sum_{i=1}^{n}\big(Y_i - \hat\beta\,(S_i - \hat\mu_S)\big),$$
where $n$ is the number of real trials and $\hat\beta$ is a coefficient fit on a separate fold of the data (cross-fitting). The real mean $E[Y]$ is the quantity we actually want. The estimator targets it without bias, for any simulator bias, because $\hat\beta$ is fit on a separate fold and multiplies a term with zero mean. The simulator only changes the variance. An empirical-Bernstein-style interval applied to the residuals $Y_i - \hat\beta S_i$ has a half-width of order
$$\sqrt{\frac{\sigma_Y^2(1-\rho^2)\,\log(1/\delta)}{n}} + \frac{c\,\log(1/\delta)}{n} + \sqrt{\frac{\hat\beta^2\sigma_S^2\log(1/\delta)}{M}}.$$
Here $\sigma_Y^2$ and $\sigma_S^2$ are the real and simulated outcome variances, $\delta$ is the allowed failure probability, and $c$ depends on the outcome range. The first term comes from the variance of the residual. The second is the usual small-sample correction. The third is the cost of estimating $\mu_S$, which is negligible because $M$ can be thousands. The number of real trials needed for a target width $\epsilon$ therefore falls from about $\sigma_Y^2/\epsilon^2$ to about $\sigma_Y^2(1-\rho^2)/\epsilon^2$, plus the lower-order term.
What is new is not the control variate itself. It is three things together. First, a finite-sample guarantee that does not require the simulator to be correct. Second, an empirical map of the per-trial correlation $\rho$ across physics regimes, which turns "the reality gap" into a quantity in units of real trials saved. Third, a test of whether simulator tuning increases that savings.
Relation to Prior Work
Two papers motivate the problem by showing that simulators are reliable for kinematics but not for contact. Both use the Kinova Mico2 arm with motion capture. Quantifying the Reality Gap in Robotic Manipulation Tasks finds that MuJoCo and Bullet predict a 90° cube rotation during pushing that never happens on the real robot. Benchmarking Simulated Robotic Manipulation Through a Real World Dataset reports a PyBullet translational error of 31,467.2 on cylinder rolling. Both report fixed-repeat error sums without confidence statements. We reuse their paired sim/real logs to estimate $\rho$ by regime.
REALM aligns 14 IsaacSim parameters to DROID Franka robots and reports $r = 0.921$ between simulated and real progression scores. That number is a correlation across policies and conditions. It does not bound the real score of any one new policy, and its own uncertainty is unstated, which is Gap 4 in the research file. We differ by using per-trial pairing and producing intervals on the real score itself.
Three papers supply ingredients or a contrast on the theory side. Provably Safe Sim-to-Real Transfer shows that real-sample cost can scale with the region where the simulator is wrong. It does this only for tabular problems where the simulator is either exactly right or wrong by at least a fixed margin. Our $1-\rho^2$ factor is a continuous analogue for evaluation rather than learning, and it needs no separability assumption. Policy Gradient for LQR with Domain Randomization bounds how many simulated systems are needed for training and reports 90% versus 10% real balance success on a Quanser Qube. It does not bound evaluation error from finite real trials. Robust System Identification gives $\tilde O(1/\sqrt T)$ rates for fitting linear dynamics. We need no model of the dynamics at all, which is why the approach applies to contact-rich tasks where that theory does not.
Traversing the Reality Gap via Simulator Tuning reduces trajectory error by 14–91% using differential evolution, and Auto-Tuned Sim-to-Real Transfer raises xArm success from 0% to 90% on Cabinet Slide. Both serve here as treatments. We ask whether lowering the mean simulation error also raises per-trial correlation, since the two need not move together.
Research Program
The program has three parts. We first state and prove the cross-fitted finite-sample interval for bounded outcomes. We extend it to graded rubrics such as REALM's Reach→Grasp→Lift→Move tiers and to binary success. For binary success, $\rho$ is capped by discreteness, so we also analyze using the simulator's continuous progress as the helper variable while the real outcome stays binary.
We then build the pairing protocol. Each real trial logs its initial object and robot state from motion capture or calibrated cameras, and the simulator is reset to that state. The size of the savings depends on this matching, so the protocol is part of the contribution.
Finally, we estimate $\rho$ and the implied number of real trials saved, separately for free-space motion, pushing, rolling, articulated objects, and deformable objects. We do this both before and after simulator tuning. The output is a table in units of real trials saved, which any lab can use to budget evaluations.
Baselines
The naive reference is the real-only mean with a Hoeffding or empirical-Bernstein interval at the same $n$ and $\delta$. It is what most papers implicitly do, and its required $n$ is the denominator of every claimed saving. We beat it if our interval at matched coverage is narrower by close to the factor $\sqrt{1-\rho^2}$ predicted from held-out $\rho$.
The strongest existing practice is sim-only scoring justified by an aggregate correlation, as in REALM, whose reported Pearson correlation is 0.921. We run it on the same policies and report how often the sim score falls outside the true real mean ± ε. That ground-truth mean comes from a large held-out pool of real trials. We beat this baseline if our intervals cover the true value at nominal rate while sim-only scoring does not, especially on contact tasks.
The closest statistical competitor is the same control-variate estimator with intervals from the central limit theorem or a bootstrap. It shares our data and compute exactly. We beat it only if its coverage falls below nominal at $n \le 30$ while ours holds. If it does not, we report that.
Two ablations isolate the key ingredients. The first replaces paired simulation from the matched initial state with simulation from randomly sampled initial states. This tests whether pairing, rather than simply having a simulator, produces the savings. The second fits $\beta$ on the same fold rather than a separate one, to show the bias that cross-fitting removes. For simulator tuning, we compare default parameters with parameters tuned by differential evolution on the identical evaluation pool. The cited paper reports 14–91% trajectory error reduction. There is no reported result on Lacuna for the per-trial sim–real correlation before or after tuning.
Why not just…?
Why not just run more real trials? This is always valid, and it is our naive baseline. But real trials are the bottleneck. Provably Safe Sim-to-Real Transfer shows how expensive simulator-free real data is even in a small gridworld. Our contribution is to say exactly how many real trials a simulator replaces. If measured $\rho$ is near zero on contact tasks, the honest answer is "none," and that is itself a useful finding.
Why not just trust the simulator once it correlates at r = 0.92? A high correlation across policies allows a constant or policy-dependent offset. It also says nothing about a new policy that exploits a solver artifact, like the phantom 90° cube rotation in Quantifying the Reality Gap in Robotic Manipulation Tasks. Our estimator stays unbiased under any such offset. The coverage comparison against sim-only scoring tests this directly.
Why not just bootstrap the control-variate estimate? This is the real threat. For moderate $n$ and well-behaved outcomes, bootstrap intervals may be nearly as tight and roughly calibrated. We settle it empirically by resampling many subsets of size $n \in \{5, 10, 20, 40\}$ from large real pools and measuring coverage and width for both methods. If the bootstrap matches our coverage, we would recommend it, and the lasting contribution becomes the pairing protocol and the $\rho$ map.
Why not just tune the simulator and skip the statistics? Tuning and our estimator work together rather than compete. Tuning lowers mean error, while our estimator benefits from correlation. Tuning that fits average trajectories could even lower $\rho$ by smoothing over trial-to-trial variation. The default-versus-tuned comparison measures which way it goes.
Proposed Experiments
The first testbed is the Quanser Qube Servo 2 from Policy Gradient for LQR with Domain Randomization. It is cheap enough to run several hundred real trials across a family of controllers of varying quality, including the domain-randomized and certainty-equivalence controllers. This pool serves as ground truth for coverage experiments. The outcomes are continuous balance time and binary success.
The second testbed is the Kinova Mico2 motion-capture data. Its existing paired logs across PyBullet, MuJoCo, and V-Rep give $\rho$ estimates per scene and per engine without new hardware, though with only about 20 repeats, so we report $\rho$ with intervals. We add new paired trials where needed.
The third testbed is REALM on DROID Franka arms. There we evaluate π0, π0-FAST, and GR00T N1.5 on a subset of skills using paired real/sim trials scored with the progression rubric. The xArm Cabinet Slide task from Auto-Tuned Sim-to-Real Transfer provides an articulated-object, before/after-tuning case.
The metrics are interval coverage at nominal 95%, interval width, the number of real trials needed to reach width ε, the measured $\rho$ by regime, and the rate of wrong policy rankings at fixed $n$.
Three checks are decisive. The theory predicts that the ratio of required real trials with and without the simulator, measured on the Qube pool, should track $1-\hat\rho^2$ estimated on a held-out split. Paired simulation should beat unpaired simulation by a clear margin. If it does not, the pairing story is wrong. Finally, for contact-rich tasks we expect $\rho$ to be markedly lower than for free-space motion, as the Kinova reality-gap results suggest. Measuring how much lower, in units of real trials saved, is the main empirical product.
References
- Quantifying the Reality Gap in Robotic Manipulation Tasks
- Benchmarking Simulated Robotic Manipulation Through a Real World Dataset
- REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
- Provably Safe Sim-to-Real Transfer
- Policy Gradient for LQR with Domain Randomization
- Robust System Identification: Finite-sample Guarantees and Connection to Regularization
- Traversing the Reality Gap via Simulator Tuning
- Auto-Tuned Sim-to-Real Transfer