Simulator Counterfactuals for Learning Safety Boundaries Without Real Failures

A robot that learns on real hardware faces a trap. To know where danger begins, it needs examples of what happens just past the edge of safety. A learner that is careful never goes there, so its estimate of that edge stays poor, and it either freezes or guesses. This proposal uses the simulator as a generator of exactly the data the real robot must never collect. At each state the real robot actually reaches, we restart the simulator from that state. There it tries the risky actions the real robot was forbidden to take, and it records whether they lead to failure. These synthetic failure labels are used only where a model trained on real transitions says the simulator agrees with reality in that neighborhood. Wherever trust is lacking, the label defaults to "unsafe."

In two sentences: the real robot gathers only safe experience, and the simulator, branched from real states, supplies the missing dangerous experience. A real-data check decides where those simulated dangers can be believed. If this works, the robot learns its safety boundary with zero real violations. It also needs fewer real samples than methods that must stay pessimistic everywhere, because it can be confident in regions where the simulator has been checked against reality.

Core Claim

The gap this targets is named in two independent results. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that as a policy becomes safer it sees fewer constraint-violating samples, so its estimate of the feasible region gets worse. Blind Spot Detection for Safe Sim-to-Real Transfer finds the same effect from the other direction: a safety classifier trained on suboptimal, boundary-visiting data reached an AUC of 0.96, against 0.60 when trained on data from an optimal oracle. Both papers motivate the proposal. Boundary data is what matters, and safe learners starve themselves of it.

Our claim is that physics-grounded simulation, branched from real states, can supply this boundary data. A residual dynamics model trained on real transitions certifies where that data is trustworthy. Here a residual dynamics model means a small learned correction added to the simulator's next-state prediction. The result should be a feasibility estimate (a learned function predicting whether a state-action pair can be kept safe) that is both accurate and not over-conservative. It should also be obtained with fewer real samples than ensemble-pessimism methods, and with no real constraint violations.

The trust rule is the one piece of formalism worth writing down. Let $\hat{y}_{\text{sim}}(s,a)$ be the simulator's safety label for taking action $a$ from real state $s$, obtained by resetting the simulator to $s$ and rolling forward. Let $d(s,a)$ be the disagreement, across an ensemble of residual dynamics models fit to real data, about how far real dynamics deviate from the simulator near $(s,a)$. Let $\tau$ be a trust threshold. The training target for the feasibility function is then

$$ y(s,a) = \begin{cases} \hat{y}_{\text{sim}}(s,a) & \text{if } d(s,a) \le \tau \\ \text{unsafe} & \text{otherwise.} \end{cases} $$

Untrusted counterfactuals are never optimistic. They collapse to the pessimistic answer. The method should therefore never be less safe than a pessimism-everywhere baseline. Its gain comes from regions where real data has confirmed the simulator, which lets it stop being pessimistic there.

How This Differs From the Closest Work

The nearest result is Provably Safe Sim-to-Real Transfer. It identifies a "mismatch region" where the simulator is wrong and plugs real transitions into that region while using the simulator elsewhere. It proves that the number of real samples needed scales with the size of that region rather than the full state-action space. That paper supplies the central intuition we reuse. It is, however, tabular. It assumes simulator errors are either zero or larger than a detectable margin, and it explores with a mixture against a known safe policy solved by linear programming.

We carry the idea to continuous control, with three changes. The hard mismatch set becomes a soft, learned disagreement score. The simulator is used as a source of counterfactual failures, not only as a dynamics model for planning. And the separability assumption is replaced by a weaker local-smoothness assumption: the discrepancy measured on the safe side of the boundary carries over to the near unsafe side. We test that assumption directly, as described in the experiments.

Two continuous-control methods constrain the design. SPiDR adds the variance of next-state predictions across randomized simulators to the safety cost. It achieves zero-shot real safety, but it is static: it never uses real data to shrink its pessimism, and it is void if reality lies outside the randomization range. SOOPER learns a probabilistic world model from real data and falls back to a safe prior policy whenever predicted cost under uncertainty exceeds the budget. It expands exploration through imagined rollouts. Its boundary knowledge, though, comes only from its own learned model, which has seen only safe data — exactly the paradox above.

We differ from both by injecting simulator-generated failure data at real states. From SPiDR we take the use of disagreement as a pessimism signal. From SOOPER we take the fallback-prior structure for safe execution.

Sim-to-Lab-to-Real trains a shield (a backup controller that overrides the main policy near failure) using Hamilton-Jacobi reachability, a method that computes the set of states from which failure can still be avoided. It then certifies the result in an intermediate lab stage that needs many physical environments. Its shield is learned in simulation only and its activation threshold is hand-tuned. Our feasibility function can play the role of that shield, but it is corrected by real data instead of lab-scale environment collection.

On the synthetic-data side, DexGrasp Anything shows the pattern of a model proposing and physics filtering, applied to millions of static grasps. ResDex shows residual policies trained on synthetic grasp proposals. Both target task success and include no safety constraints. Our generator targets the failure boundary instead.

Research Program

The system has four parts that run in a loop. First, a safe prior policy is trained in simulation with domain randomization and a Lagrangian constrained method, such as the PID-Lagrangian PPO in Safety-Gymnasium. It is made deliberately conservative using SPiDR's disagreement penalty, so it can serve as the fallback. Second, a real-robot learner explores with the fallback structure of SOOPER: it acts with its current policy and switches to the prior when the feasibility function predicts a violation. Third, after each batch of real experience, every visited real state becomes a branch point. The simulator is reset to that state, and a perturbed set of actions is rolled out, biased toward the predicted boundary, including actions the real controller refused. The rollouts are labeled by the simulator's cost function. Fourth, an ensemble of residual dynamics models is refit on all real transitions. It scores each counterfactual's trustworthiness, and the feasibility function is retrained on the gated labels from the equation above. The policy is then updated by constrained optimization against the new feasibility function.

Real states can always serve as branch points because reaching them was safe. Counterfactuals then extend only a few steps past them, which keeps the extrapolation local. Restarting the simulator requires estimating the full state from real sensing. That is straightforward for proprioceptive locomotion and RC-car tasks, and for manipulation with tracked object poses. We restrict the first study to such state-based settings and treat vision as out of scope.

The analysis question beyond "does it work" is the trust threshold. A loose $\tau$ admits simulator labels inside the mismatch region and risks optimism. A tight $\tau$ collapses to pessimism and loses the benefit. We will measure the full curve, and we will test whether a threshold set from held-out real-versus-sim prediction error, rather than tuned on outcomes, lands near the knee of that curve.

Baselines

The naive reference point is the domain-randomized prior deployed with no adaptation, trained without the disagreement penalty. On real hardware, SPiDR reports that this kind of plain randomization produced 26.1x more safety violations than SPiDR on a Unitree Go1 and 6.2x more on an RC car. It anchors the violation axis.

The strongest zero-shot competitor is SPiDR itself, run with PPO-Lag over the same randomization distribution and the same simulator compute. Its penalty weight is tuned with the same number of trials we give our threshold $\tau$. SPiDR reports 100% constraint satisfaction in the real world. Beating it means matching zero violations while reaching higher return, or reaching the same return with fewer real samples, given that it uses no real samples at all. In the sim-to-sim setting we can also place reality partly outside SPiDR's randomization range, where its guarantee is void. That is a fair stress test because our method is not told the range either.

The strongest online competitor is SOOPER, which is the closest in spirit. It gets the identical safe prior, the same real-sample budget, the same ensemble size, and the same planning compute. SOOPER reports zero violations across its simulated tasks and, on a physical RC car, a learned policy about twice as good as its safe prior in reward without a collision. Beating it means reaching its final return in clearly fewer real samples, or a higher return at an equal budget, with equal or fewer violations. Feasibility-function accuracy is the mechanism we expect to explain any gap.

For reference on the tabular regime, Provably Safe Sim-to-Real Transfer reports 0% unsafe iterations, against 68.5% for an unconstrained sim-to-real baseline, and 3.8x fewer real samples (77,958 versus 296,910) than learning without a simulator on its 5x5 windy gridworld. We will rerun that gridworld as a sanity check that our method recovers the tabular algorithm's sample advantage before scaling up.

Three ablations isolate the key ingredients. Removing the trust gate means all simulator counterfactuals are accepted. Removing counterfactuals means the feasibility function sees only real safe data plus SOOPER-style model pessimism. Replacing the simulator with the learned residual model as the counterfactual generator tests whether physics grounding matters, or whether model rollouts in the style of SOOPER's imagination suffice.

Why not just…?

Why not just use SPiDR's zero-shot pessimism and skip real-world learning? If the randomization range covers reality, this may be enough for safety, and it is a real threat on easy tasks. Its cost is permanent conservatism. The penalty never shrinks, so return stays below what is achievable. The paper itself notes that an excessive penalty weight produces policies that refuse to act, and that dynamics outside the randomization range void the bound. Our experiments settle this by reporting return at zero violations, both inside and outside the randomization support. If SPiDR matches us inside the support, we will say so and claim only the out-of-support regime.

Why not just run SOOPER with a simulator-trained prior? This is the most serious alternative, because it already explores safely with a prior and an uncertain world model. Our argument is that its feasibility knowledge is learned only from safe real data. The safety-paradox and suboptimality-paradox results above predict that its boundary estimate will be poor and its expansion slow. The "no counterfactuals" ablation is essentially this method. If it matches the full method on feasibility accuracy and real-sample efficiency, the proposal's central claim fails.

Why not just train the safety boundary entirely in simulation with domain randomization? This is cheaper and needs no simulator resets from real states. But a simulation-only boundary is wrong exactly in the mismatch region, which is where safety matters. That is the pattern behind the 26.1x violation gap reported in SPiDR. A "sim-only feasibility" variant, trained on the same counterfactual volume but from simulated rather than real branch points and with no gate, will show whether real anchoring and gating are needed.

Why not just use a large negative reward for failures? As argued by Quanyi Li et al., a fixed penalty either tolerates occasional failure or produces a robot that does nothing. In Safety-Gymnasium, PPO-Lag cut violations by 98% relative to standard PPO, at a 45% reward cost. We include penalty-PPO only as a weak reference.

Proposed Experiments

The main testbed is sim-to-sim, so that true safety labels are available everywhere. The "simulator" is a MuJoCo task from Safety-Gymnasium and the Real-World RL suite (point-goal navigation, cartpole, and walker with velocity constraints). The "real world" is the same task with planted mismatches. One is a continuous analogue of the windy cells of the provably-safe gridworld: a region with unmodeled lateral force pushing toward hazards. Another is a friction patch outside the randomization range. These planted mismatches let us compute the true feasibility region and score each method's learned feasibility function by AUC and by its false-safe rate, the fraction of truly unsafe pairs predicted safe.

The headline metrics are cumulative "real" constraint violations during learning, real samples needed to reach 90% of an oracle trained directly in the "real" environment, final return, and false-safe rate. A conservatism score, the fraction of truly safe state-action pairs marked unsafe, captures freezing.

The decisive check for local smoothness uses an adversarial mismatch that exists only on the unsafe side of the boundary, invisible from safe states. The method's prediction here is specific: the gate cannot detect this mismatch, so performance falls to SOOPER-level or worse. We want to verify that violations do not rise, because the region is reached only through branched counterfactuals the gate distrusts only if disagreement is high. If violations do rise, the smoothness assumption is load-bearing, and the paper must state it as a hard limitation.

The second decisive check is the trust-threshold sweep. Real-sample efficiency should improve as $\tau$ loosens up to a knee, and violations should stay at zero until the threshold exceeds the residual ensemble's held-out error on real data.

As a hardware step, we follow the platforms used by SOOPER and SPiDR: a 1/10-scale RC car on an obstacle course with an unmodeled low-friction surface patch. Full state is available from motion capture, which makes simulator resets exact. Success there means zero collisions, with a return above the SOOPER baseline at an equal number of real episodes.