Probes That Pay Off Later: Non-Myopic Safe Exploration for Closing the Sim-to-Real Gap

A robot trained in simulation arrives in the real world with a model that is wrong in places, and it can only find out where by moving. Moving is risky, so safe-exploration methods restrict the robot to actions that are certified safe under the model's current uncertainty. This creates a specific trap. Often no single safe probe changes which task plan the robot can certify, so a greedy probe selector sees zero benefit and stops. The useful probe is the one that shrinks uncertainty just enough to make a second, more revealing probe certifiably safe, which in turn certifies the task. We call this staircase structure safety-gated synergy. Greedy selection cannot climb such a staircase. Existing safe sim-to-real methods either avoid uncertainty entirely or probe it uniformly, so neither climbs it on purpose.

We propose to score probe sequences by how much they raise the best task performance the robot can certify. Probes are chosen by multi-step lookahead using the simulator ensemble as the belief model. A reachability shield that does not depend on the planner keeps every executed action within the real-world safety guarantee. The claim to be tested is narrow: when sim-to-real mismatch sits near constraint boundaries, lookahead over safety-gated value reaches a certified task policy with several times fewer real interactions than greedy safe probing. When mismatch is far from the boundary, it gives little benefit. The proposal is built to show both halves.

Core Claim

The myopic-query literature already documents a version of this failure in classification. Uncertainty-aware Active Learning for Optimal Bayesian Classifier shows that expected-loss-reduction learners stall when no single label flips the predicted decision. Multi-Information Source Optimization and An Information-Theoretic Framework for Unifying Active Learning Problems argue that one-step lookahead misses queries whose value only appears in combination. These papers motivate the proposal. Our observation is that safety constraints manufacture this exact structure in robotics. The set of actions a robot may take grows only as data arrives, so the information available at step two depends on what was learned at step one.

Formally, let $D_t$ be the real-world data after $t$ probes. Let $\Pi_{\text{safe}}(D_t)$ be the set of task policies certified safe given $D_t$ (by confidence-bounded reachability). Let $V(D_t)=\max_{\pi\in\Pi_{\text{safe}}(D_t)} \underline{J}(\pi; D_t)$ be the best lower-confidence task return among them. The value of a $k$-step probe sequence $a_{1:k}$ is

$$ \Delta_k(a_{1:k}) = \mathbb{E}_{\theta \sim p(\theta\mid D_t)}\big[\, V(D_{t+k}) - V(D_t) \,\big], \quad \text{subject to each } a_i \text{ being certified safe given } D_{t+i-1}, $$

where $\theta$ indexes simulator hypotheses about the real dynamics. Greedy safe probing maximizes $\Delta_1$. On a staircase of height $m$, $\Delta_1$ is zero for every safe probe until $m-1$ enabling probes have been taken. Lookahead with $k \ge m$ does not have this blind spot.

The constraint in this objective is essential. A probe is not only an information source. It also changes which future probes are allowed.

How This Differs From the Closest Work

Four papers come close, and each differs on one axis.

Provably Safe Sim-to-Real Transfer is the most direct predecessor. In tabular constrained MDPs, it certifies where the simulator is accurate, patches a mismatch region with real data, and explores with a mixture of a safe baseline policy and an exploratory policy. Its exploration is reward-free and uniform over uncertified states. We keep its safety mechanism and hybrid-model idea, but direct exploration toward the task and plan it over multiple steps. We also extend it beyond tables.

Safe Active Dynamics Learning and Control (SEELS) already plans multi-step information-gaining trajectories with guaranteed return to a safe set. It does not use a physics simulator, and it switches discretely from "reduce all uncertainty" to "exploit." Its objective is dynamics information gain, not certified task value, so it does not prefer the enabling probe over an equally informative but task-irrelevant one.

Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap shows that probing aimed at task regret beats task-agnostic identification (14g vs 22g pouring error). It has no safety mechanism, and its regret ignores which plans are certifiable. Our objective is its task-regret idea restricted to what can be certified.

ASID and Provably-Safe, Online System Identification supply the two ends of the safety–excitation tension. ASID trains an unconstrained Fisher-information probing policy. Provably-Safe Online SysID uses a safe but local receding-horizon excitation planner.

SPiDR constrains the design in the opposite direction. Penalizing ensemble disagreement achieves zero-shot safety, which means our method must beat a strong passive pessimist on task return without giving up its safety record.

Research Program

Belief model. The belief over real dynamics is a posterior over simulator parameters, drawn from the domain-randomization ensemble, plus a small residual model for unstructured mismatch. The residual follows the hybrid model of Provably Safe Sim-to-Real Transfer.

Safety shield. Safety is handled by a shield that is computed from frequentist confidence bounds on real data, not from the planner's belief. The shield uses the "return ticket" requirement of Safe Guaranteed Exploration for Non-linear Systems: every probe must keep a safe path back to a known invariant set. In the tabular case it uses the mixture-policy guarantee. This separation answers the main known weakness of lookahead: if the prior is misspecified, plans compound errors across steps. Here a wrong belief can only waste probes. It cannot cause a violation, because admissibility is decided by the shield.

Planner. We will build two versions of the lookahead planner.

Exact tree search. In tabular and low-dimensional settings, a depth-$k$ tree search over hypothetical probe outcomes is sampled from the simulator ensemble. At each node it recomputes the shield and $V$. This is the reference implementation, and it is exponential in $k$ as expected.

Amortized probe policy. For continuous control, a probe policy is trained in simulation across randomized parameter draws, in the spirit of ALINE and Amortized Safe Active Learning. Its sparse terminal reward is the gain in certified task value after the whole probing budget. Crediting the end-of-sequence gain lets the policy learn enabling probes. The shield filters the policy's actions during training and deployment, which removes the reliance on soft penalties that leaves ASAL without guarantees.

Analysis. We will characterize staircase height in terms of mismatch geometry: how far the mismatch region lies from the constraint boundary, relative to how much one probe can shrink the confidence set. This should predict in advance the environments where depth matters.

Baselines

Naive reference. The do-nothing point is domain randomization deployed zero-shot with a Lagrangian safe-RL policy (PPO-Lag from SafePO). It probes nothing. Safety Gymnasium reports that PPO-Lag gives a 98% reduction in violations relative to PPO on velocity tasks, at a 45% reward cost. Beating it means higher certified task return at equal or lower violations.

Passive safe transfer. SPiDR is the strongest passive baseline. Its paper reports 100% zero-shot constraint satisfaction on hardware and 26.1x fewer violations than domain randomization on the Go1. It will be run with the same simulator ensemble as our belief model. Beating it means matching its violation rate while achieving higher task return in environments where its pessimism blocks the goal.

Strongest active safe baseline in the tabular setting. This is the uniform certification method of Provably Safe Sim-to-Real Transfer, which reports 77,958 real samples with 0% unsafe iterations on its 5x5 windy gridworld. We will rerun it on that gridworld and on staircase variants. Beating it means reaching the same certified task value with fewer real samples while still having zero unsafe iterations.

Strongest active safe baseline in continuous control. This is SEELS-style safe multi-step information-gain probing, which reports 93.2% task success at $\delta=0.1$ on its free-flyer simulation. We will also run ASID's Fisher-information probing policy with our shield attached, to give ASID safety it does not natively have.

Fairness and ablations. Every active method shares the same simulator ensemble, the same shield, the same real-interaction budget, and the same hyperparameter-tuning budget in simulation. The key ablation holds everything fixed except lookahead depth: our objective at $k=1$, which is greedy safety-gated task value, against $k\in\{2,3,4\}$ and the amortized policy. A second ablation swaps the objective for plain information gain at equal depth. This separates the effect of looking ahead from the effect of aiming at the task.

Why not just…?

Why not just use domain randomization? It is safe only when the real system lies inside the randomized range, and it averages away the precise parameters that fine tasks need. ASID reports 0% domain-randomization success on rod balancing against 66% with active identification. SPiDR reports 26.1x and 6.2x more hardware violations under domain randomization. It remains our naive reference so that this gap is measured on our own tasks.

Why not just be pessimistic, as SPiDR is? This is a serious competitor whenever a safe low-uncertainty route to the goal exists. In those cases we expect SPiDR to match us at zero real cost, and we will report that. It fails by design when every route to the goal passes through high disagreement, because it never collects data. Our staircase environments place the goal behind exactly such a region, and the experiment measures how often SPiDR freezes there.

Why not just do greedy safe active learning with a task-weighted score? This is the most dangerous alternative, because it costs little and could suffice if staircases are rare. The $k=1$ ablation is this baseline exactly. If it matches deeper lookahead across our mismatch geometries, the paper's claim fails. We would then report a negative result about where depth matters, rather than argue around it. The classifier stagnation result in Uncertainty-aware Active Learning for Optimal Bayesian Classifier is the reason we expect it to stall on staircases, but that result comes from a different domain, so it is motivation rather than proof.

Why not just shield ASID's probing policy? A shield makes Fisher-information probing safe, but it does not make it purposeful. When the most informative motion is not yet certified, the shield clips it and nothing in the objective seeks the enabling probe. The task-oriented exploration paper also shows that task-agnostic probing wastes budget on distractors. The shielded-ASID baseline tests this directly.

Proposed Experiments

Gridworld. First, rerun the 5x5 windy gridworld of Provably Safe Sim-to-Real Transfer with its reported numbers as an anchor. Then extend it to a family of gridworlds where the goal lies behind a windy corridor next to the unsafe wall, with staircase height $m$ varied from 1 to 4 by spacing the windy cells. Exact tree search is feasible here.

The main measurement is real samples until certified task value comes within $\epsilon$ of the oracle that knows the real dynamics. We also record unsafe iterations and the fraction of runs in which each method stalls with zero improvement. The decisive prediction is that greedy safety-gated selection stalls at rates that rise sharply once $m\ge2$, while depth-$m$ lookahead does not. A control family with mismatch far from the wall should show no gain from depth.

Continuous control. Next, inject structured mismatch into Safety-Gymnasium navigation tasks (Point and Car Goal). The "real" environment uses perturbed friction and actuator gain, plus a localized force field next to hazards, which plays the role of the windy cells. We also use the kinematic-car testbed of SAGEMPC.

These tasks compare the amortized probe policy against all baselines on three measures: episodes to reach a certified policy, violations during probing, and final constrained return. Wall-clock decision latency will be reported against SAGEMPC's roughly 10 ms per solve, since amortization exists to keep lookahead real-time.

Hardware (stretch goal). As a stretch goal, we will repeat ASID's rod-balancing task on a Franka arm with an obstacle placed so that aggressive center-of-mass probes are initially uncertifiable. The comparison is against ASID's reported 66% success, with violation counts that ASID does not report.

What would falsify the claim. The project fails if the $k=1$ ablation matches lookahead on staircase environments, or if the shield's conservatism leaves every method equally stuck. Either outcome is informative, and both will be reported.