Separating Noise from Simulator Error in Safe Real-World Data Collection

A robot trained in a simulator usually needs some real-world data before it behaves well on hardware, because no physics engine gets friction, contact and actuator delays exactly right. Collecting that data is dangerous, so recent methods do it cautiously. They trust the simulator where real observations agree with it, collect real data only where they disagree, and fall back to a known-safe controller whenever they are unsure. These methods share a blind spot. They measure "disagreement" or "unsure" with a single number that mixes two different things. One is irreducible randomness in the world, such as wind gusts or wheel slip, which statisticians call aleatoric uncertainty. The other is the agent's ignorance about where the simulator is wrong, called epistemic uncertainty, which shrinks as data arrives. In a gusty or slippery region where the simulator's average behaviour is already correct, the combined number stays high forever. The agent keeps sampling there, keeps the region flagged as untrusted, and pays real-world risk for nothing. This is the robotic version of the "noisy TV" problem known from curiosity-driven exploration.

This note proposes a safe sim-to-real data-collection loop that routes the two kinds of uncertainty to different jobs. Epistemic uncertainty about the simulator's error decides where to explore and which regions count as certified. Aleatoric noise does not trigger exploration. Instead it widens the safety margin, because noise is a disturbance the controller must tolerate, not a gap it can close by looking harder. The prediction is that methods using total uncertainty spend more real samples as noise grows in regions where the simulator is accurate, while the split method's cost should stay roughly flat, with no increase in safety violations.

Core Claim

The closest work is Provably Safe Sim-to-Real Transfer. It shrinks an estimated "mismatch region" of state-action pairs where the simulator is wrong, using confidence bounds on real transitions. It then mixes a safe baseline policy with an exploratory one through a linear program, so that the real world sees no constraint violations. Its real-world sample cost provably scales with the size of the mismatch region. The method is tabular, assumes the simulator error is either zero or at least a margin $\sigma_s$, and does not separate noise from mismatch. Two continuous methods share the conflation. UARL uses total disagreement across an ensemble of critics to gate deployment and to weight adaptation data. RAMU applies a worst-case-weighted (Wang-transform) risk measure over hallucinated parameter perturbations, and explicitly leaves the epistemic/aleatoric distinction out of scope. Our claim is narrower and more testable than "add uncertainty decomposition." In a continuous safe sim-to-real loop, the simulator's error should be modelled as a smooth function whose posterior pools evidence across nearby states, with a separate learned noise term. Certification and exploration should depend only on that function's posterior, and safety margins should include the noise. Depeweg et al. supplies the precedent: separate weights on epistemic and aleatoric risk let offline policies avoid exploiting model bias without freezing in turbulent states. That work was offline and had no simulator or shield. We carry the split into online, shielded, real-world data collection.

Research Program

We model a real transition as the simulator's prediction plus a learned correction plus noise:

$$s' = f_{\text{sim}}(s,a) + g(s,a) + \varepsilon, \qquad \varepsilon \sim \mathcal{N}\big(0, \Sigma(s,a)\big).$$

Here $f_{\text{sim}}$ is the simulator's mean next state, $g$ is the unknown simulator error, and $\Sigma$ is state-dependent real-world noise. The posterior over $g$ gives an epistemic band of width $w(s,a)$, and $\Sigma$ is fitted as a heteroscedastic aleatoric head, as in the dual-headed predictors of Aleatoric Mapping Agents. A state-action pair is certified when the upper end of the epistemic band on $\|g\|$ falls below a tolerance $\tau$. Noise magnitude never enters this test.

This test is where pooling matters. A per-cell confidence interval on the mean must shrink like $\sqrt{\Sigma/n}$ from that cell's own samples. A smooth function prior on $g$ instead lets neighbouring samples inform each other. As a result, the epistemic band in a noisy but sim-accurate region tightens at the rate set by the function class, not by the local count alone. The tabular method cannot exploit this structure.

We will compare three posterior models over $g$ at equal compute: deep kernel learning, a posterior network, and a probabilistic ensemble in the style of PETS. The first two are the sampling-free estimators that Charpentier et al. found satisfy all four of their desiderata. The third is the strong, simple default. The contribution is the routing of uncertainty, not any particular estimator, so if the ensemble suffices we will say so.

Safety comes from a learned safety critic in the style of the reachability-based discounted safety Bellman equation used by Sim-to-Lab-to-Real. The critic is trained in simulation and re-evaluated on hardware under the corrected dynamics. An action is allowed only if it stays safe under two kinds of worst case at once. The first is the worst correction inside the epistemic band. The second is a high quantile of the learned noise. Otherwise a pre-trained backup policy takes over, mirroring the safe-mixture idea of the tabular paper.

Exploration then maximises the epistemic band width $w$ alone, restricted to shield-approved actions. This sends the robot toward unexplored edges of the safe region, the near-miss data that the "safety paradox" of Feasible Dual Policy Iteration says safe agents starve of. Noise at those edges does not trap it there. Once a region is certified, policy improvement uses the corrected simulator there and real data elsewhere, with FOCOPS as the constrained optimiser.

The guarantee we can honestly offer is probabilistic and conditional. It holds if the epistemic band is calibrated and the noise quantile is accurate. We will measure both kinds of calibration directly rather than assume them, and we will report a deliberately adversarial violation of the smoothness assumption.

Baselines

The naive reference point is the safe backup policy run unchanged on the "real" system, together with an unconstrained policy fine-tuned on real data. Together they bound safety from one side and return from the other. The unconstrained sim-to-real baseline in the tabular paper violated safety in 68.5% of learning iterations (Provably Safe Sim-to-Real Transfer).

The strongest directly comparable method is that tabular algorithm itself. In its 5×5 windy gridworld it reports 0% unsafe iterations and 77,958 real samples, against 296,910 without a simulator. We will rerun it on that gridworld and on variants that add stochastic but sim-accurate cells. Beating it means matching 0% unsafe iterations while needing clearly fewer real samples as the added noise grows. On the original noise-free-elsewhere gridworld we expect roughly parity and will report it as such.

For continuous domains the tabular method does not apply. We will therefore build a "total-variance twin": our full pipeline with the same posterior model, shield and optimiser, but certification and exploration driven by total predictive variance, epistemic plus aleatoric. This is the ablation that isolates the key ingredient, and the most important comparison in the note. We will tune it on the same budget, including tuning its certification threshold, so that it is not handicapped by a threshold set for our method.

We will also run UARL (UARL) with its diversity-loss critic ensemble and progressive domain randomization. No absolute reward or safety number for UARL was found on Lacuna; the file reports only that it matched or exceeded offline baselines in 84% of comparisons.

Finally, we will run RAMU (RAMU) as the strongest zero-real-data robust method. On the Real-World RL Suite it satisfied constraints in 80% of perturbed test environments, versus 51% for standard safe RL and 55% for domain randomization. Beating it means a higher fraction of constraint-satisfying test environments, or equal safety at higher return, at the cost of a reported real-sample budget.

All methods get the same simulator, the same real-interaction budget, the same backup policy where one is used, ten seeds, and the same hyperparameter-search budget.

Why not just…?

Why not just use ensemble variance, as UARL does? This is the real threat, and the total-variance twin exists to settle it. The evidence against it is indirect. In Charpentier et al., vanilla ensembles scored near chance (AUC-ROC around 0.5) at out-of-distribution detection. Total variance also stays high under noise by construction, as the noisy-TV work of Hou et al. shows for exploration. If the twin matches our sample cost across noise levels, the decomposition is unnecessary in this setting, and we will report that as the result.

Why not just randomize the simulator widely and skip real data, as RAMU or domain randomization would? RAMU's own 80% satisfaction rate leaves one in five perturbed environments unsafe. RGAT and RRLS both describe how wide randomization buys robustness with slow, cautious behaviour. Our experiments include structured mismatches outside any randomization range, where zero-data methods should fail. They also include high-noise conditions, where a worst-case-weighted risk measure should freeze the policy. If RAMU nonetheless matches us on return and safety, real-world data collection is not worth its risk on these tasks.

Why not just collect more real data with a fixed generous safety margin? A fixed margin set for the noisiest region is overly conservative everywhere else. One set for the average region is unsafe in the noisy one. The split method sets the margin locally from the learned noise. We will include a fixed-margin variant of the twin, with the margin swept, to show where on the safety-return curve each lands.

Proposed Experiments

The core study is a controlled two-by-two design on the Real-World RL Suite tasks (Cartpole Swingup, Walker Walk, Quadruped Walk) and on Safety-Gymnasium Point and Car Goal tasks (Safety Gymnasium). A held-out simulator configuration plays "reality." One factor is structured mismatch, absent or present: biased friction and actuator delay confined to part of the state space. The other is heteroscedastic noise, low or high: slip and gust noise in a different part of the state space, which the simulator reproduces correctly on average.

We report real samples to certify the mismatch-free regions and to reach 90% of oracle return, real constraint violations during collection, final return within budget, and the fraction of sim-accurate states wrongly left uncertified. We also report calibration of the epistemic band (coverage of true $g$) and of the noise quantile.

The decisive check is the noisy-but-accurate cell of the design. Our method's real-sample cost should be nearly unchanged between low and high noise. The total-variance twin should spend a growing share of its samples in the noisy region, and the rise should not be closed by retuning its threshold. A second check reverses the roles: noise is low, but the mismatch is subtle and smooth, violating the tabular paper's $\sigma_s$ separation. Here we test whether certification still avoids certifying wrong regions.

We also stress the method's own assumption by making $g$ discontinuous at a contact boundary, and report whether violations appear. Real-world transfer on a quadruped is a follow-up, conditional on the simulated decisive check succeeding.