Certified Residual Exploration: Safe Sim-to-Real Adaptation in Continuous Spaces

A robot trained in a simulator usually fails in a few specific ways on real hardware. The floor is slicker in one corner, a motor lags at high speed, a gust pushes the robot near a wall. If we knew where reality differs from the simulator, we could spend scarce and risky real-world trials only there and trust the simulator everywhere else. One paper has shown this pays off: Provably Safe Sim-to-Real Transfer proves that real-world sample cost can scale with the size of the "mismatch region" instead of the whole problem, with zero safety violations during real exploration. Its proof, however, works only on a finite grid of states. It counts visits to each grid cell, and it assumes every cell is either simulated exactly or wrong by at least a fixed margin. Neither assumption survives on a continuous robot.

We propose to model reality as the simulator plus a residual, a correction term assumed to live in a small, known family of functions. We then explore the real system only in directions that shrink uncertainty about that residual, and only through actions that remain safe for every residual still consistent with the data. The claim is that real-world sample cost and safety then depend on the dimension of the residual family, not on the number of states. The tabular result falls out as the special case where the family consists of indicator functions on grid cells.

Core Claim

Write real transitions as $s' = f_{\text{sim}}(s,a) + \theta^\top \psi(s,a) + \eta$. Here $f_{\text{sim}}$ is the simulator's next-state prediction, $\psi(s,a) \in \mathbb{R}^{d_\Delta}$ is a fixed set of residual features, $\theta$ is an unknown weight matrix with a known norm bound $\|\theta\| \le b$, and $\eta$ is bounded noise. After $n$ real transitions, regularized least squares gives an estimate $\hat\theta_n$ and a confidence set, an ellipsoid around it. Its width at a query point $(s,a)$ is

$$w_n(s,a) = \beta_n \, \|\psi(s,a)\|_{V_n^{-1}}, \qquad V_n = \lambda I + \sum_{i\le n} \psi(s_i,a_i)\psi(s_i,a_i)^\top,$$

where $\beta_n$ is the usual self-normalized concentration radius and $\lambda$ is a ridge constant. This width replaces the visit count $1/N(s,a)$ that powers tabular algorithms. It is well defined even though no continuous state is ever visited twice.

Three consequences follow. First, a direction of feature space already covered by data is trusted everywhere, including at states never visited. Second, there is no need to classify state–action pairs as matched or mismatched, so the separability assumption of Provably Safe Sim-to-Real Transfer disappears. Third, when $\psi$ is the one-hot indicator of cells in a candidate mismatch set, $d_\Delta = |B|$, and the analysis should recover that paper's dependence on the mismatch region.

We conjecture a real-world sample complexity polynomial in $d_\Delta$, horizon $H$, and $1/\epsilon$, of the same form as the linear-feature bound of low-rank SWEET in Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-Free RL, $\tilde{\mathcal O}(H^3 d^4 A^2/\epsilon^2)$. Our bound would have $d_\Delta$ in place of $d$ and no dependence on the size of the state space. The reason is that the simulator removes the need to learn anything outside the residual span. Proving the exact exponents is part of the work.

Research Program

The first component is where the residual features come from, which is where domain randomization enters in a new role. Domain randomization means training across many simulator variants with perturbed physics. Usually it is used to make a policy robust across those variants. We instead use it to learn $\psi$. We generate randomized simulator variants covering friction, mass, latency, localized force fields, and actuator gains. We record each variant's transition residual relative to the nominal simulator. We then fit a low-dimensional basis, either by principal components of the residual fields or by a small network trained to reconstruct them. The hypothesis is that the real mismatch lies near the span of these plausible mismatches, even when no single randomized variant matches reality. This is weaker than the realizability assumption in Provable Sim-to-Real Transfer in Continuous Domain with Partial Observations, which proves near-optimal transfer bounds when the real system lies in a linear-quadratic simulator class. We require only that the difference lies in a learned span. We also test directly what happens when it does not.

The second component is safety without an externally supplied safe policy. Provably Safe Sim-to-Real Transfer and SWEET both assume a known safe baseline controller $\pi^0$. We instead compute a backup policy in simulation that is robust to every residual with $\|\theta\| \le b$, together with a set of states from which that backup provably keeps the robot inside the safe region.

During real exploration, a candidate action is executed only if it passes a check. For every residual inside the current confidence ellipsoid, the next state must land where the backup can still recover. This is a pessimistic one-step reachability check. It resembles the shield in Sim-to-Lab-to-Real, which pairs a learned Hamilton–Jacobi safety critic (a value function predicting distance to collision) with a backup override. The difference is that our shield's margin comes from a data-driven residual confidence set rather than from a critic trained in a lab stage. As data accumulates, the ellipsoid shrinks and the set of permitted actions grows. That growth is how we attack the safety paradox of Breaking Safety Paradox with Feasible Dual Policy Iteration, where safer policies see fewer boundary states and therefore estimate the boundary worse. Our exploration is allowed to approach the boundary exactly as far as the residual uncertainty permits.

The third component is what to explore. The setting is reward-free, as in SWEET and the tabular sim-to-real paper: the goal is to gather enough real data that any later constrained task can be planned in the corrected simulator. Exploration policies are planned in the simulator plus $\hat\theta$. Each one maximizes the summed confidence width $w_n$ along its trajectory, subject to the shield. Exploration stops when the maximum width over the reachable safe set falls below a threshold. That threshold is tied to the target planning accuracy $\epsilon$ in the constrained objective $\max_\pi \mathbb{E}[\sum_t \gamma^t r]$ subject to $\mathbb{E}[\sum_t \gamma^t c] \le$ budget.

The fourth component is detecting that the basis is wrong. The failure mode documented in Blind Spot Detection for Safe Sim-to-Real Transfer is unmodeled features that make the agent confidently wrong. We guard against it by monitoring the normalized prediction error of real transitions against the confidence set. A statistically significant excess triggers the backup policy and expands the feature set, for example by adding local radial features at the offending states. We analyze how many such expansions can occur and what safety guarantee survives misspecification up to a known tolerance.

Baselines

The naive reference is zero-shot transfer: train with PPO-Lagrangian in the nominal simulator and deploy it unchanged. PPO-Lagrangian is PPO with a learned penalty weight on constraint cost. In Safety-Gymnasium velocity tasks it cut violations by 98% relative to PPO at a 45% reward cost (Safety Gymnasium). This baseline shows how bad the gap is before any real data is used.

The tabular reference is the mismatch-certified method of Provably Safe Sim-to-Real Transfer. On its 5×5 gridworld with windy cells it reports 0% unsafe iterations, against 68.5% for unconstrained sim-to-real. It also reports 77,958 real samples against 296,910 for learning without a simulator, roughly 3.8× fewer. On that gridworld we run our method with one-hot features and must match these numbers within noise. If we cannot, the generalization is broken. On continuous tasks we run the tabular method after discretizing states, with grid resolution tuned under the same real-sample budget.

The strongest continuous competitor is a combination the literature has not built but a reviewer would: Reinforced Grounded Action Transformation with a reachability shield added. RGAT trains an action-correcting network end to end by RL so that simulator transitions match real ones. It has no safety mechanism, so we give it the same robust backup policy and the same Hamilton–Jacobi-style override used in Sim-to-Lab-to-Real. We note that the RGAT paper gives no numerical returns; no reported result was found on Lacuna for RGAT's returns on its MuJoCo tasks. The file reports that Sim-to-Lab-to-Real's shield reduced training violations by 4% to 77% relative to the safe RL baselines SQRL and Recovery RL, and reached 76.7% success and 86.7% safety on a physical quadruped. Those figures come from a different setup and are context, not a target we can match directly.

All methods get the same simulator, the same randomized-variant budget, the same real-transition budget, and the same hyperparameter search budget, which we fix in advance and count in environment steps. Beating the strongest competitor means two things. We need fewer total real violations, ideally zero, at equal final constrained return. We also need to reach within $\epsilon$ of the true-environment constrained optimum using fewer real samples.

Two ablations isolate the key ingredient. The first, full-dynamics uncertainty, uses the same shield and exploration rule but places confidence sets on the entire dynamics with no simulator prior. It is a continuous analogue of SWEET and tests whether the simulator is doing the work. The second, random residual basis, replaces the domain-randomization-learned $\psi$ with random Fourier features of equal dimension. It tests whether learning the basis matters.

Why not just…?

Why not just use wide domain randomization and skip real exploration? Randomization is a real threat when the true mismatch sits inside the randomized range. The file notes two limits. Wide randomization makes policies conservative in contact-rich tasks. And Robust Gymnasium shows sharp drops under structured disturbances, with frequency mattering as much as magnitude. Randomization also gives no certificate against violations. Our experiments include a condition where the real mismatch is a spatially localized force field that randomization covers only in expectation. We report the full return-versus-violations frontier as randomization width varies. If some width yields zero-shot policies that match ours on both axes, we will say so and conclude that real exploration is unnecessary in that regime.

Why not just discretize and run the tabular algorithm? We run exactly that as a baseline. We expect its sample cost to grow exponentially with state dimension, because grid cells multiply and each needs its own visits. Hashing heuristics like the SimHash buckets in Reinforcement Learning from Safety Feedback mislabel safe states that share a bucket with unsafe ones. The decisive plot is real samples to reach $\epsilon$ against state dimension, on Point (2D navigation), Car, and Ant variants.

Why not just learn a safe dynamics model from real data from scratch? This is the full-dynamics ablation. If it matches our method, the simulator prior adds nothing and the proposal fails its central claim. SWEET's low-rank bound suggests the relevant dimension would then be that of the full dynamics, which is much larger than $d_\Delta$ when the gap is localized.

Why not just use PPO-Lagrangian with real-world fine-tuning? Lagrangian methods satisfy the constraint only at convergence and oscillate during training. The file notes that PID-controlled multipliers are needed to damp this. We include PID-Lagrangian fine-tuning (CPPO-PID from SafePO) and count every real violation.

Proposed Experiments

The surrogate "real" environments are modified simulators, following the practice in RGAT and the tabular sim-to-real paper. In Safety-Gymnasium Point-Goal, Car-Goal, and Ant velocity tasks, we inject three kinds of mismatch. The first is a localized wind field near hazards, the continuous analogue of the windy cells. The second is a friction patch. The third is actuator lag, applied through Robust Gymnasium's environment disruptor.

Metrics are cumulative real violations during exploration, real samples until the constrained policy planned in the corrected simulator is within $\epsilon$ of optimal, and final normalized reward and cost.

Four checks decide the proposal. The first is the gridworld reproduction with one-hot features. The second is scaling: sample cost should track $d_\Delta$ and stay flat as irrelevant state dimensions are added, while the discretized baseline grows. The third is the full-dynamics and random-basis ablations. The fourth is a deliberate misspecification test, a mismatch orthogonal to the learned basis. There we measure whether the residual-excess detector fires before any violation, and how many violations occur when it does not. If the tested methods show near-boundary coverage improving together with zero violations, that is direct evidence against the safety paradox. If violations appear under misspecification, we report the tolerance at which the guarantee breaks.