Certified Shields That Grow With Real Data: Infinite-Horizon Invariants Indexed by the Sim-to-Real Gap

A robot trained in simulation can be given a formal safety proof that holds forever, but only for the simulator. Once it meets real hardware, unmodeled friction, delay, or mass errors break the proof. The usual fix is to assume a pessimistic worst-case error and prove safety against that. This leaves the robot penned into a small, timid region. We propose a different arrangement. Before deployment, in simulation, we compute a whole family of safe regions, each proven safe forever under a different assumed size of simulator error. Each region is an inductive invariant: a set of states the controller can never leave once inside. During real-world learning, the robot measures how far reality departs from the simulator in each part of the state space. It then moves to the larger proven region that the measured error allows. The safe region grows as evidence accumulates, and no formal verification runs online.

The core claim is that the sim-to-real gap can be treated as a parameter of the safety proof rather than as a reason to abandon it. Doing so turns three separate problems into one loop: proving safety over an unbounded horizon, exploring without crashing, and transferring from simulation. In that loop, verification happens offline against the simulator. Real-world data is used only to certify which precomputed proof currently applies. Exploration is the process of earning larger certified regions. The guarantee we can honestly offer has two parts. One is a formal infinite-horizon part: invariance holds whenever simulator error stays within the certified bound. The other is a statistical part: a calibrated, monitored estimate that the bound is valid. We keep the two parts explicitly separate rather than blending them.

What exists and what is missing

Three bodies of work frame the gap. On the verification side, Verifying Reinforcement Learning up to Infinity builds inductive invariants for ReLU controllers from template polyhedra, which are sets bounded by faces in fixed chosen directions, solved with mixed-integer linear programs. Compositional Inductive Invariant Based Verification of Neural Network Controlled Systems splits each invariance check into two parts linked by a "bridge predicate". A network part is solved by bound propagation, and an environment part is solved by an SMT solver (a logic-based constraint solver such as Z3). This scales to networks of 2×1024 neurons in seconds. Provably Safe Neural Network Controllers via Differential Dynamic Logic proves infinite-time safety by showing that the network stays inside a proven control envelope. All three motivate this proposal and also constrain it: each proves safety only for an exact nominal model, and none says what happens when that model is wrong.

On the exploration side, REVEL achieves zero violations during training by shielding a neural policy with a verified symbolic policy and an inductive "safe bubble". It assumes fixed known worst-case dynamics, and its re-verification consumes about 87% of training time. Adaptive Probabilistic Shielding re-synthesizes shields as estimated transition intervals shrink. It argues that unshielded exploration is necessary to escape a "conservativeness trap". This is a negative result we must confront directly, although it is established only for discrete models with known topology. ReCORS uses conformal prediction to bound unknown next states without distributional assumptions. Conformal prediction is a calibration method that gives coverage guarantees from held-out residuals. ReCORS supplies the statistical tool we reuse, but it reasons only over finite horizons of up to 50 steps.

On the transfer side, Provably Safe Sim-to-Real Transfer isolates a "mismatch region" where the simulator is wrong and shows that real sample cost scales with that region's size. It is tabular and uses discounted cost, not invariance. SPiDR penalizes safety cost by disagreement across randomized simulators but freezes the policy after simulation. Sim-to-Lab-to-Real gives PAC-Bayes certificates (probabilistic generalization bounds) and still saw violations in 13.3% of hardware trials. Optimized Control Invariance Conditions is the closest work. It filters actions on a physical quadrotor using a control barrier function, with residual dynamics learned by a neural ensemble. It does not use a simulator, has no exploration loop, and relies on Gaussian-style mean-minus-k-sigma bounds.

What none of these do is precompute infinite-horizon invariants as a function of the size of the gap, and then use real data to select among them. That is the new piece.

Core mechanism

Let $f_{\text{sim}}(x,u)$ be the simulator's next state, and let real transitions be $x' = f_{\text{sim}}(x,u) + w$, where the residual $w$ is unknown. We partition the state space into cells $c$. For an increasing set of residual levels $\delta_1 < \dots < \delta_K$ and a backup controller $\pi_b$, we verify offline a family of sets $\mathcal{I}_{\boldsymbol\delta}$. Here $\boldsymbol\delta$ assigns one residual level $\delta(c)$ to each cell. Each set must satisfy

$$ x \in \mathcal{I}_{\boldsymbol\delta} \;\wedge\; \|w\|_\infty \le \delta(c(x)) \;\Rightarrow\; f_{\text{sim}}(x,\pi_b(x)) + w \in \mathcal{I}_{\boldsymbol\delta}, $$

where $c(x)$ is the cell containing $x$. Each set must also lie inside the safe set. Smaller certified residuals admit larger invariants.

The compositional decomposition of Zhou & Tripakis makes this cheap. The residual enters only the environment half of the check, so the bridge predicate on the network half is computed once. Only the SMT environment check is repeated for each residual level. The polyhedral templates of Bacci et al. serve as the invariant shape class. Because the family is ordered, we search it monotonically rather than enumerating all $K^{|\text{cells}|}$ assignments. We grow one cell's level at a time from the uniform pessimistic assignment and verify only the assignments the online loop could actually request.

Online, an exploratory learning policy acts freely. A runtime monitor replaces its action with $\pi_b$ whenever the successor state could leave the currently selected invariant under the certified residual bound. Every real transition yields a residual $x'_{\text{real}} - f_{\text{sim}}(x,u)$, which is logged to its cell. Per-cell split-conformal quantiles produce residual bounds with stated coverage. The PAC one-step style analysis of Wu et al. provides an alternative with a confidence parameter. When a cell's certified bound drops below a smaller $\delta_k$, the shield switches to the larger precomputed invariant. If an observed residual exceeds its cell's certified bound, the monitor falls back to the most robust invariant and marks the cell as part of the mismatch region. This is the continuous analogue of the region $B$ in Provably Safe Sim-to-Real Transfer.

The growth argument against the conservativeness trap is geometric. Tighter bounds on cells already inside the current invariant license a larger invariant. That larger invariant contains new frontier cells that the agent can now visit safely and measure. Unlike the discrete setting of Brorholt et al., continuity lets interior evidence enlarge the boundary. Whether this escapes the trap without unshielded steps is an empirical question we test directly.

Research Program

The work has three stages. First, we build the offline verifier for gap-indexed invariant families on top of compositional checking, and measure how verification time scales with $K$ and with the number of cells. Second, we build the online certification loop with conformal per-cell bounds and the fallback monitor. We then characterize the guarantee formally. Invariance is unconditional given the bound. The probability that the bound fails is controlled per step by calibration, so over $T$ steps the failure probability scales at most linearly with $T$ by a union bound. The fallback monitor converts detected failures into recoverable events, provided the robust invariant retains margin. Third, we stress the loop with delay and latency, the failure mode that Dulac-Arnold et al. identify as catastrophic. We model delay by augmenting the state with recent actions and ask whether residual bounds on the augmented system remain informative.

Baselines

The naive reference point is a fixed pessimistic robust invariant. It is verified once against a single residual bound covering the entire randomization range and is never updated. This baseline isolates whether online certification buys anything.

The strongest competitor is the residual-learning barrier-function filter of Optimized Control Invariance Conditions. It is the only prior method that keeps a forever-invariant set while learning the real error from data on hardware. We give it the same real transitions, the same backup controller, and its ensemble-based mean-minus-k-sigma bound, with k tuned on the same calibration split. No reported result was found on Lacuna for its violation rate or certified-region size on our benchmarks. We will run it ourselves.

Two other strong baselines cover the remaining design space. REVEL is given the nominal simulator as its dynamics model. It reports zero training violations on its 10 benchmarks when the model is correct, and we expect it to fail once reality differs. SPiDR is trained on the same randomization range, with $\lambda$ tuned on the same budget. No reported numerical result was found on Lacuna for SPiDR on these benchmarks.

For context on the probabilistic route, Sim-to-Lab-to-Real reports 86.7% hardware safety on quadruped navigation. On the verification-scale side, Bacci et al. verify 20 of 22 Adaptive Cruise Control agents and 11 of 11 Bouncing Ball agents under nominal dynamics.

We include two ablations of the key ingredient. One replaces conformal bounds with ensemble bounds, which separates our contribution from the closest competitor's statistics. The other keeps conformal bounds but uses one global residual level instead of per-cell levels, which tests whether spatial localization of the gap matters.

Beating the strongest baseline means one of two outcomes at equal real-data budget. The first is a larger certified region with no more real violations. The second is fewer violations at equal certified region size. In both cases, total online verification time must be lower than REVEL's re-verification.

Why not just…?

Why not just verify once against a worst-case bound from the randomization range? Our pessimistic baseline is exactly this. Its weakness is the conservatism that the file documents for domain randomization: one bound sized for the worst corner of parameter space shrinks the invariant everywhere. If its certified region and task return match ours on most benchmarks, then the online loop is not worth its complexity, and we would report that. We expect a gap where the error is spatially concentrated, such as contact or saturation regions.

Why not just learn a residual model and filter with a barrier function, as the quadrotor work does? This is the real threat. It already achieves forever-invariance under learned error on hardware. The differences are testable. It has no simulator prior, so it must learn the full residual rather than the deviation from a good simulator. It has no mechanism for safely enlarging its region. Its Gaussian-style bounds can be miscalibrated under the heavy-tailed or delay-induced residuals that Dulac-Arnold et al. highlight. The ensemble-versus-conformal ablation measures calibration directly. If ensembles stay calibrated and grow the region as fast as our method, our contribution reduces to offline precomputation, and we will say so.

Why not just re-verify REVEL-style each time the dynamics estimate changes? Re-verification is what costs REVEL 87% of its training time. It also requires the online model to be a sound worst-case bound, which it is not across a reality gap. We will report wall-clock verification time per accepted region update for both approaches. Our claim fails if incremental re-verification on our benchmarks costs the same as our offline family.

Proposed Experiments

We will use the verification benchmarks where infinite-horizon proofs already close: Adaptive Cruise Control, Bouncing Ball, and Cartpole from Bacci et al., plus the 2D maze from Zhou & Tripakis. On ACC and Cartpole we also compare with ReCORS's 50-step Cartpole horizon, where standard safe baselines failed beyond 20 steps. Our goal there is infinite-horizon invariance rather than a longer finite horizon.

The "real" system follows the surrogate protocol of RGAT. We hold out perturbed physics, including mass, friction, and actuator gain, and add the action and observation delays of the realworldrl-suite. Some perturbations are placed deliberately outside the randomization support to probe the regime where SPiDR's pessimism collapses. A final stage on a small quadrotor would mirror the hardware setting of Optimized Control Invariance Conditions.

The metrics are:

  • real-world constraint violations per run during learning;
  • certified invariant volume over real-data budget;
  • task return;
  • fallback-trigger rate;
  • empirical coverage of the per-cell bounds;
  • offline and online verification compute.

Three checks decide the outcome:

  1. Certified volume must grow monotonically with real data while violations stay at zero, with no unshielded steps. This directly tests the conservativeness-trap claim.
  2. Under out-of-support perturbations, the fallback monitor must detect the excess residuals and prevent violations that REVEL and SPiDR incur.
  3. The per-cell conformal bounds must beat the ensemble ablation on calibrated coverage under delay. If that fails, the statistical half of our guarantee is no better than the closest prior method.