Spending Real-World Risk Only Where the Simulator Is Wrong

A robot that learns by copying a human demonstrator in simulation will usually drift when moved onto real hardware. Its small mistakes push it into situations the demonstrations never covered, and in those situations its mistakes grow. The standard fix is to let the robot drift and have an expert relabel the states it lands in. On a physical robot, that drift is exactly what breaks hardware.

This proposal starts from one observation. The simulator is a free, tireless, perfectly safe source of corrective labels everywhere it agrees with reality. Real-world risk and human attention only need to be spent in the places where the simulator is wrong. We propose a continuous-control method that does three things. It estimates those "mismatch regions" from a small amount of real data. It uses the simulator, corrected by a learned residual model, as the expert that supplies recovery actions inside trusted regions. It spends an explicit real-world safety budget only on probing and correcting behavior inside untrusted regions. The testable claim is that real-world constraint violations, real samples, and human queries should scale with the size of the mismatch region near the demonstrated behavior, not with the size or length of the task.

Core Claim

Compounding error in imitation learning means that one-step mistakes accumulate quadratically over a horizon when the learner never sees recovery data, as formalized in Cornelisse et al.. After sim-to-real transfer, that drift is not uniform. It concentrates where simulated and real dynamics disagree, for example around unmodeled contact, a localized disturbance, or a sticky surface. Inside a simulator we can already fight drift cheaply, by perturbing the robot and planning a recovery back to the demonstrations. What we cannot do in simulation is know where those recoveries will be wrong on the real system.

We therefore treat sim-to-real imitation as a constrained Markov decision process, meaning a control problem with a separate cost signal whose expected total must stay under a budget $\epsilon$. We add a learned partition of state-action space into a certified region, where the simulator plus residual model is trusted to label recoveries, and a mismatch region. In the mismatch region the robot either falls back to a safe backup behavior or spends budget gathering real data. What would be new is the combination of three things in continuous spaces with neural function approximation. The first is simulator-as-expert relabeling in the style of DAgger, gated by certified dynamics agreement. The second is a cost-constrained recovery planner. The third is a budget-allocation rule that sends deliberate, bounded exploration toward the mismatch boundary instead of away from it.

Where This Sits in Prior Work

The motivating result is Provably Safe Sim-to-Real Transfer. That paper shrinks an estimated set of state-action pairs where simulator and reality differ. It builds a hybrid model, with real counts on the mismatch set and simulator transitions elsewhere, and it explores with a mixture of a safe baseline and an exploratory policy. It proves that real-world sample complexity scales with the mismatch set's size. On a 5×5 windy gridworld it reports 0% unsafe iterations versus 68.5% for an unconstrained baseline, and 77,958 versus 296,910 real samples.

We borrow its structure directly: a mismatch set, a hybrid model, and a budget-respecting mixture. We change three things it leaves open. It is tabular, and we work in continuous control. It assumes the shift is either zero or separated by a detectable margin, and we replace that assumption with calibrated uncertainty on a residual dynamics model. It is reward-free with no demonstrations, and we make the expert demonstrations the object being transferred.

Two imitation methods supply the recovery machinery. SAILOR searches over action corrections inside a learned latent world model with Model-Predictive Path Integral control, a sampling-based planner that perturbs action sequences and keeps the best ones. It then distills successful recoveries back into a diffusion policy, and it beats a 10× larger demonstration set. However, it learns its world model from scratch, was evaluated only in simulation, and its search is unconstrained. ThriftyDAgger gates human interventions by novelty (disagreement across a policy ensemble) and risk (a learned task-completion value). It reports 73% autonomous success on simulated peg insertion versus 48% for LazyDAgger and 24% for SafeDAgger. However, a human is its only source of corrections, and its risk score measures task failure, not a safety cost.

Our planner is SAILOR's, with three changes: it runs on a simulator corrected by a residual model rather than a latent model, it is restricted to certified regions, and it carries a cost constraint. Our human gate is ThriftyDAgger's, but it fires only in mismatch regions where the simulator cannot answer.

Safety components come from three sources. Backward Value Functions estimate cost already spent in an episode, so that a whole-episode budget becomes a per-state check. We use them to set the mixing weight online. Sim-to-Lab-to-Real pairs a reachability-based backup policy with a performance policy and relabels intended actions when the shield overrides. We reuse the backup-plus-relabeling pattern, but inflate the backup's safety margin by the local mismatch estimate. Proactive projection shows that projecting risky actions onto the safe set cuts emergency interventions roughly tenfold (2.5% to 0.25% of steps). This tells us to prefer projected rather than abrupt fallbacks.

Two results constrain the design. Blind Spot Detection found that blind-spot classifiers trained on data from an optimal oracle reach AUC 0.60 versus 0.96 from a suboptimal one, because optimal behavior never visits the boundary. Feasible Dual Policy Iteration reports the same starvation for safety critics. Both imply that a purely conservative explorer will never learn where the simulator is wrong. That is why our budget is spent on deliberate probing near the mismatch boundary rather than held in reserve.

Research Program

The method has three stages.

Stage 1: simulation. We train a diffusion policy from a small demonstration set. We also build a simulator recovery expert. From perturbed states near the demonstrations, a cost-constrained planner running on true simulator dynamics finds actions that return to the demonstrated states. We aggregate these labels DAgger-style until the policy recovers robustly in simulation. In the same stage we train a cost critic and a backup policy.

Stage 2: real-world mismatch estimation. In the target ("real") environment, every executed transition updates an ensemble of residual dynamics models, each predicting the gap between real and simulated next states. A state-action pair is certified when a calibrated upper bound on that gap, $u(s,a)$, falls below a tolerance $\tau$. The calibration uses split conformal prediction on held-out real transitions, and we will report where its exchangeability assumption is stressed by policy-induced data. The tolerance $\tau$ is set from the cost critic's margin, so that certified errors cannot flip a safe state to an unsafe one.

Stage 3: execution and learning. When the ensemble-novelty gate signals drift, the recovery planner searches over action corrections using the simulator plus residual model. It rejects any imagined trajectory that leaves the certified region or exceeds the cost critic's limit. The accepted recovery is executed, and its label is added to the training set. If no certified recovery exists, the robot hands control either to the margin-inflated backup or to a human.

Exploration toward uncertified states, which is what shrinks the mismatch region, is done by mixing a probing policy $\pi_{\text{probe}}$ with the backup $\pi_{\text{safe}}$. The probing policy is chosen to target high-$u$ states near the demonstrations. At each step, the probing weight $\alpha_t$ is the largest value satisfying

$$C_{\le t}(s_t) + \alpha_t\, \hat{C}_{\text{probe}}(s_t) + (1-\alpha_t)\, \hat{C}_{\text{safe}}(s_t) \le \epsilon .$$

Here $C_{\le t}$ is the backward-value estimate of cost already incurred, $\hat{C}_{\text{probe}}$ and $\hat{C}_{\text{safe}}$ are forward cost estimates for each policy (inflated by $u$), and $\epsilon$ is the episode safety budget. This is the continuous, per-state analogue of the tabular mixture in Provably Safe Sim-to-Real Transfer.

The prediction that real-world cost scales with the mismatch region's size comes from that paper's bound. Our analysis goal is a weaker continuous version: the number of uncertified visits is bounded by a covering number of the mismatch region near the demonstrations.

The testbeds are all simulator-as-reality surrogates, the same protocol RGAT uses, because the file offers no shared physical benchmark. The first is a continuous re-creation of the windy-wall gridworld in Safety-Gymnasium Point-Goal and Point-Button. There, "reality" adds localized force fields near hazards, and the cost is hazard contact.

The second is RoboMimic and RoboSuite manipulation (Lift, Can, Square, Peg Insertion), as used by SAILOR and ThriftyDAgger. In those tasks, "reality" changes friction and mass only in a sub-region of the workspace, or adds actuator latency, or adds an unmodeled compliant fixture. The cost covers contact-force and joint-velocity limits.

We vary mismatch type deliberately. A global parametric shift, such as friction scaled everywhere, should favor domain randomization. A localized shift, such as a windy patch or a sticky zone, should favor our method. Unmodeled-state shifts in the spirit of Blind Spot Detection should defeat every simulator-based method unless a human is consulted.

Baselines

The naive reference is the sim-trained diffusion policy deployed zero-shot with no real-world adaptation. It sets the floor for success and the ceiling for violations.

The strongest imitation competitor is ThriftyDAgger run directly in the surrogate-real environment. A scripted expert stands in for the human, and the method gets the same demonstrations, the same simulator-pretrained initialization, and the same human-query budget we allow. Its reported 73% autonomous success on simulated RoboSuite peg insertion is the reference number. We will reproduce it on the unshifted task before shifting dynamics, and we count our method as beating it only if it matches or exceeds that autonomous success under shift while using fewer expert queries and fewer cost violations.

The strongest recovery-search competitor is SAILOR, with its latent world model trained on the same real rollouts we collect. We run it in two forms: unconstrained, and with a reactive shield bolted on. Both forms get the same planning compute and the same number of real transitions. No reported per-task result for SAILOR was found on Lacuna in the summary text, so its numbers come from our own reproduction. We claim a win only if our method reaches equal success with fewer violations during learning, not merely at convergence.

The strongest safe sim-to-real competitor is the tabular algorithm of Provably Safe Sim-to-Real Transfer, run on a discretized version of our point-navigation surrogate. We first reproduce its gridworld figures (0% unsafe iterations, 77,958 versus 296,910 real samples) as a correctness check of our implementation. We expect to match its safety and to beat its real-sample count once discretization makes the state space large.

Two further baselines complete the set. One is domain randomization over the same physics parameters, with an identical tuning budget for the randomization ranges. The other is PPO-Lagrangian fine-tuning in the real surrogate starting from the imitation policy; Safety-Gymnasium reports a 98% violation reduction with a 45% reward drop for that method relative to PPO.

The ablations isolate the key ingredient, which is targeting by mismatch. The first replaces the certified-region gate with uniform trust in the simulator plus residual model. The second spends the same exploration budget uniformly rather than on high-$u$ states. The third removes the simulator expert, so that every off-manifold label comes from the human. The fourth removes the cost constraint from the recovery planner. If uniform budget spending matches targeted spending, the central claim fails.

Why not just…?

Why not just randomize the simulator's physics and transfer zero-shot? For global, parametric mismatch this is a serious threat, and we expect it to be competitive or better in that condition. That is exactly why the experiments include a global-friction shift. The file's evidence against it is narrower but real. Randomization forces conservatism across a wide envelope, which hurts precision contact tasks. RGAT reports that grounded transformation outperforms action-noise envelopes on Hopper and HalfCheetah. And Blind Spot Detection shows that randomizing parameters of a model cannot expose state that is missing from the model entirely. The localized and unmodeled-shift conditions will show whether our advantage exists beyond the setting where randomization is the right tool.

Why not just run ThriftyDAgger on the real robot with a safety shield? This uses the human for every off-manifold correction, including corrections the simulator could have provided for free. Our prediction is that human queries under ThriftyDAgger scale with how often the policy drifts, while ours scale with how often it drifts into mismatch regions. The no-simulator-expert ablation is precisely this alternative, run at equal budget. If query counts come out equal, the simulator-as-expert idea adds nothing.

Why not just ground the whole simulator with RGAT first, then do imitation and recovery entirely in the grounded simulator? This is the most plausible rival. It differs from our method mainly in where real data are collected: everywhere, rather than where the simulator is wrong. RGAT itself collects real transitions with no safety mechanism, so we compare against a version whose grounding data are gathered by the same budgeted mixture policy, at an equal real-sample budget. If global grounding matches our violation and sample counts when mismatch is localized, then targeting is unnecessary, and we would reframe the contribution as safe grounding alone.

Why not just add a large collision penalty to the recovery planner's objective? Quanyi Li et al. document the failure mode. Small penalties are traded away for reward, while large ones produce a robot that freezes. The penalty also does nothing about the planner trusting a wrong model. The unconstrained-planner ablation, with a tuned penalty, tests this directly.

Proposed Experiments and Decisive Checks

The decisive experiment fixes the task and the demonstrations and varies only the area of the localized mismatch region, from zero up to most of the reachable workspace, in both the point-navigation and peg-insertion surrogates. The primary metrics are three: cumulative cost violations during real-world learning, real transitions until success plateaus, and expert queries. Final autonomous success rate and episode cost relative to $\epsilon$ are secondary.

The claim survives if all three primary quantities grow roughly in proportion to mismatch area for our method, while for ThriftyDAgger, SAILOR, and uniform-budget exploration they stay tied to task length or total drift. At zero mismatch area, our method should need almost no real-world intervention. At full-workspace mismatch, it should converge toward ThriftyDAgger-with-shield, since nothing is certified.

A second check targets the suboptimality paradox. We will measure the AUC of our mismatch estimator against ground-truth mismatch, comparing targeted probing with conservative, backup-only data collection. If the result echoes the 0.96-versus-0.60 gap from Blind Spot Detection while violations stay under budget, that supports spending risk deliberately.

A third check stresses calibration. We will report how often certified transitions actually exceed $\tau$ under the policy-induced data shift, because a certificate that is violated more often than its nominal rate would undermine the whole safety argument.