A safety shield is a supervisor that sits between a learning robot and its motors and overrides any action that could lead to a crash. Shields are how people let a reinforcement learning agent explore on real hardware. Every shield makes its decision from what the sensors report, and in the real world the sensors are wrong: LiDAR (laser range sensors) misreads reflective surfaces, cameras blur, tactile arrays drift. A shield that trusts a noisy reading lets the robot into danger. A shield that hedges against the worst possible reading blocks the narrow safe paths that lead to a rarely seen goal, so an agent that is only rewarded on success never sees a success. This note proposes making the shield's margin of distrust, meaning how far the true state may lie from the reported one, a quantity the robot learns online. It learns this margin with a no-regret algorithm that watches its own sensor errors once a cleaner state estimate arrives a few steps later. Because the margin is learned from sensor residuals and not from crashes, setting it does not cost any safety violations.
The opportunity is that three literatures each hold one piece. Safe exploration methods that guarantee few violations during learning assume clean states. Methods that are robust to sensor noise either inject noise during training or plan against a fixed worst case. Adversarial online learning offers tools that track a moving target with guarantees even when the target is chosen by an adversary, as time-varying, correlated sensor noise effectively is. Put together, they address a failure that none of them handles alone. The shield's distrust margin controls both how often the robot violates constraints during exploration and whether a sparse reward is ever found, and the right margin changes as the noise changes.
Core Claim
We claim that when a reachability shield is run on noisy observations, its intervention test should ask whether any state within a learned radius of the estimate is unsafe. The radius should be set per region of the state space by an online learner that receives delayed feedback, namely the gap between the raw observation and a smoothed or redundant state estimate that becomes available $d$ steps later. Concretely, the shield overrides the agent's action at time $t$ when
$$\min_{s \in \mathcal{B}(\hat s_t,\, r_t(z_t))} V(s, a_t) < 0,$$
where $\hat s_t$ is the state estimate from the current observation, $\mathcal{B}(\hat s_t, r)$ is the ball of radius $r$ around it, $z_t$ is a coarse context (for example, the region of the map or the sensor's reported confidence), $V$ is a safety value function learned in simulation whose negative values mark states from which a crash cannot be avoided, and $r_t(z)$ is the learned radius for context $z$. The radius for each context is updated from the delayed residual $e_{t-d} = \lVert s_{t-d} - \hat s_{t-d} \rVert$ by online gradient steps on the quantile ("pinball") loss at target miss rate $\alpha$. By a standard online quantile-tracking argument, the long-run fraction of steps on which the true state falls outside the ball converges to $\alpha$ for any noise sequence, including adversarial ones, with an error term that grows with the delay $d$. When the true state is inside the ball and $V$ is accurate, the override is sound. The violation rate is therefore bounded by the miss rate plus the error in $V$. Meanwhile the radius shrinks wherever the sensors are good, and that reopens the corridors that a fixed worst-case radius closes.
The new element is not robust shielding itself, and it is not online quantile tracking by itself. It is the closed loop between the two, together with the claim that this loop resolves the deadlock between conservative safety and sparse rewards under noise that is non-stationary and depends on the state.
Research Program
The work builds on and departs from several specific prior results. In the motivating evidence, Robust Gymnasium builds a modular "observation disruptor" and shows that PPO and SAC collapse under moderate observation noise, and that persistent low-magnitude noise is more damaging than isolated large shocks. It is a benchmark and supplies no method; we use its disruptor as our noise source. Revisiting Safe Exploration shows that cumulative cost hides runs of consecutive violations and introduces the Expected Maximum Consecutive Cost (EMCC) metric. A miscalibrated shield under correlated noise is exactly the kind of system that would produce such runs, so we report EMCC.
The work that supplies the method comes from shielding and adversarial online learning. Sim-to-Lab-to-Real trains a Hamilton-Jacobi reachability critic and backup policy in simulation and uses it as a runtime shield on a quadruped. It notes that the shield threshold must be hand-tuned and that conservative thresholds cause freezing. We reuse its shield structure and replace the hand-tuned threshold with a learned, state-dependent perception margin. The idea of tuning a safety-relevant hyperparameter with an adversarial bandit comes from Non-stationary CMDPs, whose bandit-over-bandit layer runs Exp3 over candidate variation budgets. Its feedback signal is reward and constraint violations, so learning the parameter costs violations. Ours learns from sensor residuals, which arrive whether or not anything went wrong. Masoudian et al. show that delayed feedback inflates adversarial regret but can be handled without knowing the maximum delay. This both predicts and limits how our miss-rate guarantee degrades as the smoothing lag $d$ grows, and we test that prediction directly.
Some prior work constrains the design. SAC-RIS learns a robust invariant set against a worst-case disturbance and reaches zero deployment violations, but it requires violations while the set is being learned and fixes the adversary's strength in advance. We therefore confine all learning of $V$ to simulation and treat the disturbance strength as something to estimate, not something to assume. Backward Value Functions documents how conservative state-wise bounds keep agents out of high-reward regions near the constraint boundary. Our sparse-reward experiments are designed to reproduce this deadlock and measure how much of it the learned radius removes. Provably Safe Sim-to-Real Transfer gives tabular guarantees that require clean states and a separable sim-to-real shift. We do not claim to extend its theory to continuous noise. Instead, we offer an empirical continuous-state counterpart for the perception side of the gap.
Three things will be built. The first is the shield, with the inner minimization over the ball computed by sampling plus a projected-gradient refinement on $V$. The second is the per-context radius learner, along with a variant that weights a grid of fixed radii using an adversarial experts algorithm, so we can check that the result does not depend on which online method is used. The third is a sparse-reward exploration agent: PPO receiving reward only at goal arrival, with action relabeling as in Sim-to-Lab-to-Real so it learns from the actions it intended to take. The analysis will cover the coverage bound under delay, along with a characterization of when context-dependent radii matter, namely when noise magnitude depends on the state.
Baselines
The naive reference is a nominal shield that applies the same simulation-trained $V$ directly to the noisy estimate with radius zero. It shows how much damage sensor noise does when nothing is done. A second, cheap reference is a Lagrangian method with no shield. We use PPO-Lag, for which Safety-Gymnasium reports a 98% reduction in safety violations relative to PPO at the cost of only a 45% drop in total reward on velocity-constrained tasks. This anchors what penalty-based safety buys on clean observations. Lagrangian methods only enforce the constraint on average and at convergence, so we expect it to be weak on violations during training, but it is the standard comparison.
The strongest existing competitor for worst-case robustness is a fixed-radius robust shield in the spirit of SAC-RIS, with the radius set to the largest noise magnitude in the disruptor schedule. SAC-RIS reports zero constraint violations under learned worst-case disturbances on Cart-pole and Quadrotor. No reported result was found on Lacuna for SAC-RIS under observation noise or sparse rewards. To beat it, we must match its violation rate within a tolerance we will pre-register and achieve significantly higher goal-reach rate on sparse tasks.
The strongest simple competitor is noise-injection training, which Robust Gymnasium argues is essential: the shield and policy are trained in simulation under the full noise distribution, then deployed with a nominal shield. The violation-driven tuner adapted from Wei et al., which runs Exp3 over the same radius grid but receives feedback only from violations and returns, isolates whether the residual feedback channel matters. A fixed calibrated radius, set once to the $(1-\alpha)$ quantile of residuals collected during a warm-up period, isolates whether the online adaptivity matters. An oracle that knows the instantaneous noise level gives an upper reference.
All methods share the same simulation-trained $V$ and backup policy wherever they use a shield, the same PPO exploration agent, the same environment-step budget, the same noise schedules, and the same tuning budget for hyperparameters. We use twenty configurations per method and select on held-out noise seeds.
Why not just…?
Why not just train with injected noise, as Robust Gymnasium recommends? This works if deployment noise resembles training noise. The cases we target are noise whose level shifts during deployment and noise that depends on where the robot is. A policy trained on the average noise level is either too cautious where the sensors are clean or too trusting where they are bad. We test this directly, with injection matched to deployment as one condition and mismatched as another. If injection matches our method under mismatch, our claim fails.
Why not just use a large fixed radius and accept the conservatism? The deadlock documented by Backward Value Functions suggests that worst-case margins close narrow corridors to sparse goals. Under sparse reward, that means no learning signal at all. The SafetyPointGoal corridor variants will show whether goal-reach rate actually collapses at the worst-case radius. If it does not, the sparse-reward half of our argument is weak.
Why not just calibrate the radius once from a warm-up and keep it fixed? This is the real threat. Under stationary noise it should perform about as well as our method, and we expect and will report that outcome. The learned radius only earns its place under drift, under correlated or adversarial noise, or under noise that depends on the state. The decisive comparison is therefore on non-stationary disruptor schedules, where a fixed quantile's miss rate should exceed $\alpha$ after a shift and stay there.
Why not just filter the sensors (Kalman filter or autoencoder, as in Prakash et al.) and shield the filtered state? Filtering reduces error but does not tell the shield how large the remaining error is, and the filter itself lags. Filtering is complementary: we run every method on filtered estimates as well, and the radius learner then tracks the filter's residual error.
Proposed Experiments
The main benchmarks are Safety-Gymnasium's SafetyPointGoal1-v0, with reward given only on goal arrival and added hazard corridors of varying width; Circle2D at all four difficulty levels, scored by EMCC; and the safe-control-gym 2D quadrotor used by SAC-RIS. Observations on all three are corrupted through Robust Gymnasium's observation disruptor with four noise schedules: stationary Gaussian, slow drift, abrupt level switches, and noise whose magnitude depends on the state, concentrated near hazards (emulating reflective walls). We add an adversarial disruptor that pushes the estimate toward the safe side of the boundary. The delayed ground truth is the simulator state revealed after $d \in \{0, 5, 20, 50\}$ steps, with a second version in which the delayed estimate comes from a smoother that is itself noisy.
The metrics are violations during learning (cumulative and EMCC), the empirical coverage miss rate compared with the target $\alpha$, goal-reach rate and steps to first success under sparse reward, and the fraction of steps on which the shield intervenes.
The decisive checks are these. First, on schedules with level switches or state-dependent noise, our method should beat the fixed calibrated radius on violations at equal goal-reach rate. Second, our method should beat the worst-case radius on goal-reach rate at comparable violations. Third, the miss rate should stay near $\alpha$ under the adversarial disruptor, with degradation as $d$ grows that is consistent with delayed-feedback regret. Fourth, the violation-driven Exp3 tuner should incur more violations during learning than residual-driven tuning. If the fixed calibrated radius matches us on non-stationary noise, or if worst-case shielding does not deadlock sparse goals, we will report that the learned margin is unnecessary in these regimes.
References
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning
- Revisiting Safe Exploration in Safe Reinforcement learning
- Sim-to-Lab-to-Real: Safe reinforcement learning with shielding and generalization guarantees
- Provably Efficient Model-Free Algorithms for Non-stationary CMDPs
- A Best-of-both-worlds Algorithm for Bandits with Delayed Feedback with Robustness to Excessive Delays
- Robust Safe Reinforcement Learning under Adversarial Disturbances
- Constrained Markov Decision Processes via Backward Value Functions
- Provably Safe Sim-to-Real Transfer
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- On the use of Deep Autoencoders for Efficient Embedded Reinforcement Learning