Stop Probing When the Answer Can No Longer Change: Budget-Coupled Stopping for Safe Sim-to-Real Transfer

A robot trained in simulation usually has to touch the real world before it can be trusted there. It pushes an object, lifts a cup, or drives near a wall to learn where the simulator was wrong. These probes are the most dangerous part of transfer: the robot deliberately goes where its model is uncertain, and that is often where contact, drops, and collisions happen. Current methods either probe for a fixed amount of time or keep probing until they are confident about every place the simulator might be wrong. Both waste physical risk on questions that do not affect the task.

We propose a simple change in what the stopping rule asks. The robot should not ask "do I know the real dynamics well enough?" It should ask "could one more probe change which policy I deploy, or whether I can certify that policy as safe, by enough to justify the safety risk of the probe itself?" We make probing and deployment draw on one shared safety budget. Every unit of constraint cost spent on probing is unavailable for deployment, so the rule for when to stop sensing follows from the same constrained optimization that governs safe exploration. The claim to test is that this budget-coupled, decision-focused stopping rule gets the safety of certified safe transfer while using fewer real samples than confidence thresholds over the whole mismatch region and fewer than the best fixed probing budget. The advantage should be largest when the size of the sim-to-real mismatch varies from one deployment instance to the next.

Core Claim

A constrained Markov decision process (CMDP) is a reinforcement learning problem in which the agent maximizes reward while keeping the expected total of a separate "cost" signal, such as collisions or excess force, below a budget $\epsilon$. We treat the whole transfer episode as one CMDP. It has a probing phase, in which the robot gathers real transitions to correct the simulator, and a deployment phase, in which it executes the task. After $t$ probes the robot holds data $D_t$ and a posterior over how the real dynamics differ from simulation. From this posterior it computes the deployment policy $\pi^*(D_t)$ with the highest expected return among policies whose posterior-certified cost fits in the remaining budget $\epsilon - C_t$. Here $C_t$ is the cost already spent on probing. Let $V(D_t)$ be that policy's expected return. The robot stops probing at the first $t$ for which

$$\mathbb{E}_{o \sim p(\cdot \mid D_t)}\big[V(D_t \cup o)\big] - V(D_t) \;\le\; \mathbb{E}\big[V \mid \text{budget reduced by } c_{t+1}\big] - V(D_t) + \eta,$$

where $o$ is the outcome of the next planned probe, $c_{t+1}$ is its predicted constraint cost, and $\eta \ge 0$ is a small tolerance for model error. In words, the robot stops when the expected improvement in the deployed policy from one more observation is no larger than the loss caused by spending more of the safety budget.

This rule has two properties that neither purely informational nor fixed-horizon stopping has. First, uncertainty about dynamics that no candidate deployment policy visits has zero value, so the robot never probes it. Second, the stopping threshold tightens by itself as probing spends budget, which follows the "cost-so-far" logic of Constrained Markov Decision Processes via Backward Value Functions.

Where This Sits

Several papers set up the gap. Provably Safe Sim-to-Real Transfer is the closest work. In tabular CMDPs, it certifies a mismatch region $B$ where simulator and reality disagree. It explores that region with a mixture of an exploratory policy and a known safe policy, and proves that sample complexity scales with $|B|$. It keeps sampling until confidence thresholds hold across all of $B$, whether or not those states matter for the task. We keep its safe-mixture probing but replace uniform certification with the decision-focused rule above, and we move to continuous states.

Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap trains probing policies to reduce downstream task regret rather than parameter error. This supports our choice of target. However, its probes have no safety constraint and run for a fixed horizon.

Evo-NeRF shows that stopping camera acquisition once grasp confidence is high saves about 41% of sensing time on a real robot. That is evidence that task-confidence stopping pays off physically, but it involves no constraints and no dynamics model.

Combining Simulations and Real-robot Experiments for Bayesian Optimization of Bipedal Gait Stabilization is the one prior system that touches all three directions. It stops multi-fidelity entropy search when expected entropy change plateaus and uses simulation as a heuristic filter against falls. It works over only four gait parameters, its stopping signal is informational rather than decision-focused, and its safety is not certified.

Other papers supply the method. Safe Exploration Using Bayesian World Models and Log-Barrier Optimization supplies the ensemble world model for epistemic uncertainty and the log-barrier optimizer that keeps training nearly violation-free. We reuse both, and add the transfer setting and the stopping rule that it lacks. Information-Theoretic Safe Bayesian Optimization supplies the idea of choosing queries by mutual information about the safety boundary rather than about the objective. We apply it to choose which probe to run next.

Two results constrain the design. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safe policies starve their safety estimates of boundary data. Blind Spot Detection for Safe Sim-to-Real Transfer finds a similar effect: optimal, fully safe demonstrations give a danger classifier with AUC 0.60, while suboptimal probing data gives 0.96. Together these warn that a stopping rule driven only by the deployment policy's own visitation will stop too early, because it has never seen the boundary. Our probes are therefore chosen to be informative about the constraint boundary of candidate policies, not just their typical trajectories.

Research Program

The system has four components.

The first is a mismatch model. It is an ensemble of residual dynamics models, each predicting the difference between real and simulated transitions. It starts at zero residual and is trained on real probe data. We use a recurrent state-space ensemble for image observations, following CERL, and a small MLP ensemble for state observations.

The second is a candidate deployment set. From simulation we train a finite family of policies spanning the trade-off between reward and cost, for example with different Lagrange multipliers. This makes $\pi^*(D_t)$ a selection among candidates plus a short fine-tune, rather than a full re-solve at every probe. This is the main computational lever. Without it, estimating the expected value of the next probe would need nested policy optimization, the cost that makes Liang et al.'s pipeline heavy.

The third is the probe selector. It proposes short real-world probe trajectories and scores each by its mutual information with the event "the chosen candidate violates the remaining budget under the true dynamics." A shield rejects any probe whose worst-case cost across the ensemble would exceed the per-probe allowance. The shield uses the ensemble's pessimistic rollout and falls back to a known safe policy, in the style of the mixture policy in Provably Safe Sim-to-Real Transfer.

The fourth is the stopping test in the equation above. We estimate it by sampling outcomes from the ensemble and re-scoring the candidate set.

The main technical risk is false convergence: an overconfident ensemble reports that nothing is left to learn. We address this in two ways. The tolerance $\eta$ is calibrated in sim-to-sim runs so that the rate of "stopped, then deployment violated" is held below a target. We also report how that rate varies with ensemble size. In the tabular case we will also check whether the rule inherits the $|B|$-scaling guarantee restricted to the task-relevant part of $B$. If it does, that becomes a theoretical result. If it does not, we will report where the argument breaks.

Baselines

The naive reference point is zero-shot transfer with no probing. A policy is trained in simulation under domain randomization over the same parameter ranges our ensemble is initialized with. It uses the same simulator samples and the same hyperparameter-search budget as our candidate family, and it is deployed directly. This reference shows how much probing is worth at all.

The strongest existing method on the question of safe probing is the certified mismatch-region approach of Provably Safe Sim-to-Real Transfer. It stops only once confidence holds throughout $B$. On its 5x5 windy gridworld it reports 0% unsafe iterations, compared with 68.5% for unconstrained sim-to-real, and 77,958 real samples, compared with 296,910 for reward-free safe RL without a simulator. We will run it on exactly that benchmark using the authors' published setup. In our continuous tasks we will run it through the same ensemble mismatch model, so the only difference is the stopping and probe-selection rule. Beating it means matching 0% unsafe iterations with fewer real samples. We expect the gap to grow as a larger share of $B$ lies off the task's paths.

The strongest task-oriented competitor is the regret-minimizing probing of Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap. On physical pouring it reports 14g average error, compared with 22g for task-agnostic exploration. We run it with its fixed horizon and give it the same shield we use, so that the comparison isolates stopping rather than safety. Beating it means equal or lower task error with fewer probes and no more constraint cost.

Because a well-chosen fixed budget is the most dangerous simple competitor, we also run a fixed-budget oracle. It uses our full system but stops after $T$ probes, with $T$ tuned in hindsight separately per task on the test distribution. This gives it more information than any deployable method has. Our method beats it only if it wins on distributions where the needed amount of probing varies between instances.

Three ablations isolate the key ingredients. The first keeps everything but stops on the entropy of the mismatch posterior, a task-agnostic threshold in the style of the coverage-map and multi-fidelity entropy rules (Exploring Unknown Environments with Mobile Robots using Coverage Maps). The second keeps the decision-focused value but removes the budget coupling: probing cost is not deducted from deployment. The third selects probes by information about the deployed policy's return rather than its constraint boundary. This tests the safety-paradox prediction.

All methods receive the same simulator samples, the same real-sample cap, the same ensemble architecture, and the same hyperparameter-search budget.

Why not just…?

Why not just use domain randomization and skip probing entirely? Randomizing over a wide range forces the policy to be safe in the worst case of that range, which is known to make contact-rich behavior sluggish. Liang et al. report 22g pouring error for task-agnostic approaches against 14g for task-oriented exploration. Randomization also gives no certificate for the real instance. Still, on instances where the real dynamics fall well inside the randomized range, zero-shot transfer may be close to optimal. Our rule should then stop after zero or one probe. We will report the fraction of instances where it does so. If our method rarely beats domain randomization on task return at equal safety, the probing half of the proposal fails.

Why not just probe for a fixed, well-tuned number of steps? This is the real threat. If every instance needs roughly the same amount of probing, a tuned $T$ will do about as well as any adaptive rule with no overhead. The case for adaptive stopping rests on variation between instances, which Evo-NeRF's 41% average saving suggests is common in practice. We test it directly. We sweep the spread of mismatch magnitudes across instances, from uniform to heavy-tailed, and compare against the hindsight-tuned fixed-$T$ oracle. We expect no advantage at zero spread and a growing one as spread increases. If the advantage does not appear even at high spread, the adaptive rule is not worth its complexity.

Why not just stop when the dynamics posterior is confident enough? Uncertainty about the dynamics treats every state equally. The robot keeps probing mismatches the task never visits, which is exactly the inefficiency of certifying all of $B$. It can also stop early in flat, uninformative regions. The entropy-stopping ablation measures this directly, and the gridworld variant with task-irrelevant windy cells is built to expose it.

Why not skip probing and deploy a policy that is robust to the whole prior? That is our method stopped at $t=0$, and the rule can choose it. When the prior's worst case does not constrain the best candidate policy, the rule should do so, and the comparison shows how often the budget spent on probing buys back task return.

Proposed Experiments

The first experiment reproduces the tabular setting of Provably Safe Sim-to-Real Transfer. We use its 5x5 windy gridworld and add variants where some windy cells lie off every candidate policy's path. The decisive check is whether we keep 0% unsafe iterations while needing fewer than the reported 77,958 real samples. The saving should grow with the fraction of task-irrelevant mismatch.

The second experiment is sim-to-sim in Safety Gymnasium, where a perturbed copy of each environment stands in for reality. We use the Goal, Button, and Push navigation tasks and the velocity-constrained locomotion tasks. Perturbations to friction, mass, and actuator gain are drawn per instance with controllable spread. We measure normalized reward, normalized cost including probing cost, real probe steps, and the rate of false stops. Safety Gymnasium reports that PPO-Lag reduces violations by 98% at a 45% reward drop relative to PPO on velocity tasks. This anchors the scale of the reward-cost trade-off we expect in our candidate family. We also run the image-based Safety-Gym Point tasks used by CERL, where the ensemble must operate on images. CERL reports near-zero accumulated cost there but no reward numbers, so no reported reward result was found for that benchmark.

The third experiment is physical. We use the Franka pouring task of Liang et al. and add a constraint on spilled mass and end-effector force. It is run with a varied set of cups whose fill levels and masses differ. We compare pouring error against the reported 14g, bearing in mind that our constrained setup is harder, and compare probe count and constraint cost against their fixed-horizon method under the same shield.

Across all three settings, the result that would settle the question is this. The budget-coupled, decision-focused rule should lie on a better frontier of real samples against task return, at matched constraint cost, than both the hindsight-tuned fixed budget and the entropy-threshold ablation, and it should do so when mismatch spread is high. At zero spread its advantage should disappear, which is what the theory predicts.