Robots trained in simulation fail on hardware for two related reasons. Planners exploit errors in their own world model, and the simulator itself is wrong in places nobody anticipated. One line of work handles the first problem by forcing the planner to search only inside a learned low-dimensional "manifold" of physically valid motions: a compressed coordinate system, usually the latent space of a generative model such as a variational autoencoder, in which every point decodes to a plausible configuration. A second line handles the second problem by finding where the simulator disagrees with reality and exploring only there, while staying within a safety budget. The second line has provable guarantees, but only for small tabular worlds and only when simulator errors are either zero or large.
This note proposes joining the two. The planner's latent manifold also becomes the coordinate system for estimating simulator error. The robot starts from a conservative sim-trained policy. It measures how real transitions differ from simulated ones, attaches a calibrated error bar to each region of the manifold, and plans only through regions where predicted cost plus error bar fits the safety budget. It explores just beyond that safe region, toward places where the error bar is wide but a known-safe fallback is still reachable. The bet is that sim-to-real error in continuous robots varies smoothly along the low-dimensional manifold of feasible motion, even though it looks unstructured in raw state-action space. If that holds, a handful of real trajectories should be enough to map it, and the planner cannot exploit the error model off-manifold, because it never goes there.
Core Claim
The closest existing work splits cleanly along the gap this proposal targets.
Provably Safe Sim-to-Real Transfer supplies the central idea. It isolates the "mismatch region" where real transitions deviate from the simulator, gathers real data only there with a safe mixture policy, and reuses simulator data everywhere else. On a 5x5 windy gridworld it reports 0% unsafe iterations and 77,958 real samples, against 296,910 for simulator-free safe exploration. Its theory, however, is tabular and assumes a "separable shift": simulator error is either zero or at least some margin. Continuous friction, backlash and compliance errors violate that assumption.
SPiDR is the strongest continuous-space competitor. It adds the disagreement of an ensemble of randomized physics models to the cost and reports 100% real-world constraint satisfaction on a Unitree Go1, where plain domain randomization produced 26.1x more violations. SPiDR is zero-shot and static, though. By its own stated limitation, it cannot see dynamics outside the randomization range, where ensemble disagreement is near zero.
Two latent-space planners supply the manifold machinery. Reaching Through Latent Space plans Franka Panda paths by gradient descent in a VAE latent space, using a prior loss and learned collision classifiers. First Steps does the same for ANYmal poses; there, 100% of latent-planned trajectories respected joint velocity limits, versus nearly 60% of raw-space trajectories requiring impossible motor speeds. Both freeze the manifold offline and never adapt to physics that differ from training.
What would be new is a continuous, sample-efficient version of mismatch isolation. It replaces the separability assumption with calibrated, latent-indexed error bounds, and it is run by a manifold-constrained planner. The claim is specifically that the manifold earns its place twice: once as a regularizer on planning, and once as the space in which simulator error is learnable from few real samples.
Research Program
The method has three stages, and each mostly reuses existing pieces.
In simulation, with domain randomization, we train three things. The first is a generative model of feasible short trajectory segments; its latent code $z$ is the planning variable. The second is a latent dynamics and cost model. The third is a continuous safety margin, in the style of the Hamilton-Jacobi reachability critics used by Sim-to-Lab-to-Real and Safe Flow Q-Learning. That critic predicts distance to failure rather than a binary failure label. We also keep a backup policy trained SPiDR-style, so that the starting point is already as safe as current zero-shot methods.
On the real system, every executed segment yields a residual: the gap between the simulated and observed outcome of the planned latent segment. That gap is measured as safety margin and next-state error. We fit a small residual model $\hat r(z)$ over latent coordinates and wrap it with a conformal bound. Conformal prediction is a distribution-free way to turn held-out errors into an error bar with a stated coverage rate, following the calibration recipe of Safe Flow Q-Learning. This gives $\delta(z)$, an error bar on the margin that holds with probability $1-\beta$ over calibration data.
The planner then solves
$$\max_{z_{1:H}} \sum_{t=1}^{H} \hat R(z_t) \quad \text{s.t.} \quad \hat m_{\text{sim}}(z_t) + \hat r(z_t) - \delta(z_t) \ge 0 \ \text{ and } \ \log p(z_t) \ge \tau \ \text{ for all } t.$$
Here $z_{1:H}$ is a sequence of $H$ latent segments, $\hat R$ is predicted reward, $\hat m_{\text{sim}}$ is the sim-trained safety margin, $\hat r$ and $\delta$ are the residual correction and its error bar, $p$ is the generative model's latent prior, and $\tau$ is a density threshold that keeps the plan on-manifold.
Exploration adds a bonus on $\delta(z)$, but only for segments from which the backup policy's certified safe set remains reachable. This is the continuous analogue of the safe mixture policy. As real data accumulate, $\delta$ shrinks where the simulator is right, which recovers performance that pessimistic methods give up. Where the simulator is wrong, $\hat r$ corrects the margin.
Using continuous margins addresses the "safety paradox" of Breaking Safety Paradox with Feasible Dual Policy Iteration. A safe agent stops observing violations and its cost critics decay. Here, every transition, safe or not, gives a residual on the margin, so the error map keeps receiving data.
Three analyses accompany the method. The first is a residual-smoothness study: does simulator error have lower sample complexity to fit in latent coordinates than in raw coordinates? We measure this as held-out residual error versus number of real segments. The second is a latent-dimension sweep. GoldenStart found 1–2 latent dimensions optimal for its regularizing effect, while the latent planners note that very small latent spaces lose precision. The third is an audit of conformal coverage under the distribution shift that exploration itself causes. Exploration breaks the exchangeability that conformal guarantees assume, so we report empirical coverage per round rather than claiming the nominal rate.
Baselines
The naive reference is plain domain randomization deployed zero-shot with a Lagrangian safe-RL policy. It shows how bad an unaware transfer is; SPiDR reports this reference with 26.1x more hardware violations on Go1. It gets the same simulator, randomization ranges and training compute as everything else.
The strongest existing method for zero-shot safety is SPiDR, at 100% real-world constraint satisfaction on Go1 and the RC car. For a fair comparison we run it with the same ensemble size (at least 8, where the authors report negligible overhead) and the same randomization ranges. We tune its pessimism weight $\lambda$ with the same grid budget we give our margin threshold. We also allow SPiDR to be retrained between real rounds with randomization ranges refit to the real data. Beating it means two things. On in-support mismatch, we must match its zero-violation record while reaching higher task return within a fixed real-sample budget. On out-of-support mismatch, we must incur strictly fewer cumulative violations.
The strongest adaptation baseline is shielded fine-tuning in the style of Sim-to-Lab-to-Real: a reachability shield with backup override, fine-tuning in raw action space under the same real-sample budget. It isolates whether online adaptation alone, without manifold planning, explains any gain.
On the gridworld, the reference point is Provably Safe Sim-to-Real Transfer itself, at 0% unsafe and 77,958 samples. There we expect to tie on safety and do not expect to beat its sample count, since the tabular method is purpose-built for that setting.
Two ablations isolate the key ingredient. The first keeps everything identical but fits the residual model and plans in raw state-action space, with ensemble disagreement replacing the conformal bound. The second keeps latent planning but removes residual learning, so the planner trusts the sim margin plus a fixed buffer. Equal compute and tuning budgets apply to all of them.
Why not just…?
Why not just widen SPiDR's randomization ranges? This is the most serious threat. If the real mismatch can be anticipated, wider ranges might deliver the safety we deliver without any real-world loop. SPiDR's own stated limitation is that the true dynamics must lie inside the randomized support, and its authors also note that strong pessimism leads to freezing. Widening ranges therefore trades violations for conservatism. Our in-support condition tests exactly this. If widened SPiDR reaches the same return as our method with zero violations there, the contribution reduces to the out-of-support case. We would then present it that way rather than claim a general win.
Why not just learn a residual dynamics model in raw state space with an uncertainty ensemble? This is standard practice and needs no generative model. Its known failure is the one documented by Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future and Prediction and Control with Temporal Segment Models: optimizers drive raw-space plans into regions where the learned model, here the residual, is wrong. First Steps shows the analogous gap for feasibility. The raw-space ablation settles this directly. If it matches our violations and sample efficiency, the manifold is unnecessary and the proposal fails its core claim.
Why not just fine-tune with a reachability shield, as Sim-to-Lab-to-Real does? Shielding is reactive. It overrides actions near the boundary but does not tell the robot where to collect data, so many real trajectories may be needed. Sim-to-Lab-to-Real itself depends on a physical Lab stage with many fine-tuning rollouts and still reports an 86.7% safety rate on hardware. The comparison at equal real-sample budgets will show whether directed exploration of the error map buys fewer violations, or only fewer samples.
Proposed Experiments
Sanity check. We first reproduce the windy 5x5 gridworld. There the latent space is trivial, and the method should behave like tabular mismatch isolation: 0% unsafe, with a sample count near 77,958.
Main suite. The main suite uses Safety Gymnasium velocity and goal tasks, including PointGoal2, which SPiDR uses (no reported result found on Lacuna for SPiDR's PointGoal2 numbers). We build a "sim/real" split. "Sim" is the nominal model with domain randomization. "Real" is a perturbed model in one of two conditions. The in-support condition has friction and mass shifts inside the randomization range. The out-of-support condition has added actuator delay, ground compliance and payload, none of them randomized. Metrics are cumulative constraint violations over the whole adaptation period, real samples needed to reach 90% of an oracle trained on the "real" model, final return, and the fraction of episodes in which the agent stalls without completing the task. Stalling measures over-conservatism.
Decisive checks. Three outcomes would decide the question. Out-of-support, SPiDR's ensemble disagreement should stay near zero on the shifted dynamics while it violates constraints, and our residual bound should catch the shift within a few rounds. The latent residual model should reach a given held-out error with clearly fewer real segments than the raw-space ablation; if it does not, the smoothness hypothesis is false. Empirical conformal coverage should stay near nominal across exploration rounds; if it drops sharply, the safety argument needs a shift-aware correction before any hardware test.
Hardware. Only if all three checks pass do we move to a quadruped hardware trial matching SPiDR's Go1 protocol, with an added unmodeled payload.
References
- Provably Safe Sim-to-Real Transfer
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Reaching Through Latent Space: From Joint Statistics to Path Planning in Manipulation
- First Steps: Latent-Space Control with Semantic Constraints for Quadruped Locomotion
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies and Conformal Prediction
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies
- Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future.
- Prediction and Control with Temporal Segment Models
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark