Randomization Ranges as Safety Specifications for Pixel-Based Shields

Robots trained in simulation are made robust by domain randomization: the simulator varies lighting, sensor noise, friction and mass during training so that the real world looks like one more random draw. Formal safety tools work differently. A control barrier function is a learned or hand-built function of the robot's state that a verifier proves the controller can keep non-negative, which keeps the robot out of unsafe states. These tools assume the controller knows its state exactly. On a camera-driven robot that assumption fails first. A glare or a noisy frame shifts the estimated state, and the safety filter, which we call a shield, then approves an action that is unsafe for the true state.

This note proposes using the randomization ranges themselves as the formal specification. Whatever photometric and additive sensor noise the simulator is allowed to inject defines a set of perturbations. At every control step we compute a certified bound on how far the perception network's state estimate can move over that set. The barrier check is then run on the worst state within that bound plus a calibrated nominal error, rather than on the point estimate. The online bound must be computed on every frame at control rate, which is where partial convolution comes in. Partial convolution is an efficient convolutional layer that spatially convolves only a fraction of the channels and passes the rest through unchanged. We test the hypothesis that its identity channels make per-frame bound propagation cheaper and tighter. If that hypothesis fails, the rest of the proposal still stands.

Core Claim

A shield driven by pixels can carry a guarantee that holds for every frame and every sensor perturbation inside the randomization spec. The price is a safety margin that adapts to each input instead of a fixed worst-case buffer. Concretely, let $f$ be the perception encoder mapping an image $x$ to an estimated low-dimensional state $\hat s = f(x)$. Let $T_\theta$ be a parametric sensor transformation (brightness, contrast, gamma, bounded additive noise, small translations) with parameters $\theta$ in the randomization box $\Theta$. Define the per-frame sensitivity radius

$$r(x) = \max_{\theta \in \Theta} \lVert f(T_\theta(x)) - f(x) \rVert_\infty ,$$

which we upper-bound by linear relaxation (bound propagation through the network, as in the LiRPA and interval methods used by Exact Verification of ReLU Neural Control Barrier Functions). Suppose the transformation class is closed under inversion, meaning the clean frame is $T_\theta(x)$ for some $\theta\in\Theta$. Then the clean-frame estimate lies within $r(x)$ of $f(x)$. A residual error $e_0$ between clean-frame estimate and true state is then calibrated statistically on randomized simulation data. The shield accepts action $u$ only if

$$\min_{\lVert s-\hat s\rVert_\infty \le e_0 + r(x),\; p \in P} \Big[\nabla b(s)^\top F_p(s,u) + \alpha\big(b(s)\big)\Big] \ge 0 .$$

Here $b$ is the barrier, $F_p$ the dynamics with physical parameters $p$ in the dynamics-randomization box $P$, and $\alpha$ a class-K function, i.e. an increasing function that is zero at zero. The guarantee therefore has two explicit parts. One is formal, covering perturbations inside $\Theta$ and parameters inside $P$. The other is probabilistic, covering $e_0$.

Sensor corruptions that cannot be inverted, such as motion blur and beam dropout, are not covered by the formal term. We fold them into the calibrated term and report them separately.

Relation to Prior Work

Two papers motivate the gap. Verification-Guided Shielding for Deep Reinforcement Learning partitions the state space offline with Marabou into safe and unsafe regions and wakes an online shield only near unsafe ones. It assumes clean 7-beam LiDAR readings, so a perception error can keep the shield from ever triggering. State-wise safe reinforcement learning with pixel observations learns a barrier on the latent space of an image autoencoder. That barrier is purely empirical, and a corrupted frame can map an unsafe state into a "safe" code. We differ from both. Our barrier lives on physical state, where it can actually be verified, and perception error enters the check as a certified, frame-specific interval.

Two lines of work supply the methods. For the barrier side, Exact Verification of ReLU Neural Control Barrier Functions verifies ReLU barriers up to 8-dimensional systems. We reuse its machinery and append the dynamics parameters $p$ as constant state dimensions, so the certificate covers the whole dynamics-randomization box. For the margin side, Verification of neural reachable tubes via scenario optimization and conformal prediction shows that scenario and conformal bounds recover verifiable volume where exact methods fail. We use the same statistical tools for $e_0$, but only for the residual that cannot be certified formally.

Two papers constrain the design. Provably Safe Sim-to-Real Transfer and Provable Sim-to-real Transfer in Continuous Domain with Partial Observations give genuine guarantees, but only for tabular systems and linear-Gaussian systems respectively. This tells us end-to-end proofs for nonlinear pixel control are out of reach. That is why we split the guarantee into a formal sensor term and a calibrated residual instead of claiming a full transfer theorem.

The randomization side comes from Sim-to-Real: Learning Agile Locomotion For Quadruped Robots for dynamics and (CAD)$^2$RL for visuals. Both treat randomization ranges as training heuristics. We make the same ranges the object of verification.

Partial convolution comes from Run, Don't Walk, whose FasterNet runs 2.8–3.3× faster than MobileViT-XXS. FreeNet treats the bypassed channels as a free residual path. Both were evaluated only on ImageNet and COCO. A caution from MUXConv applies here: channel slicing can add runtime overhead without hand-tuned kernels. We therefore measure wall-clock bound-propagation time rather than counting operations.

Research Program

The system has three parts.

The first is a perception encoder from 64×64 camera images (or LiDAR scans) to the barrier's state variables, meaning robot pose, velocity and relative obstacle positions. It is trained under the full randomization distribution. We build it in three matched-accuracy variants: a FasterNet-style partial-convolution encoder, a depthwise-separable encoder, and a plain small convolutional encoder.

The second is a ReLU barrier on the low-dimensional state. It is verified offline over the dynamics box $P$ using the exact-verification pipeline above.

The third is an online shield. At each step it computes $r(x)$ by bound propagation over $\Theta$, adds $e_0$, and solves the robust barrier condition. That condition is a small linear program, because $\nabla b$ is piecewise constant on ReLU activation regions. If the condition fails, the shield substitutes a backup action.

Two scientific questions sit inside this system. The first is how conservative the margin is. If $r(x)$ is loose, the shield blocks everything and the robot never reaches its goal. We will measure the margin's size against the empirically realized worst-case error over sampled $\theta$. We will also train with a bound-tightening regularizer, interval-aware training on $\Theta$, and measure how much task reward it costs.

The second is whether partial convolution's architecture matters for certification. In a partial convolution layer, the bypassed channels' bounds pass through exactly, and only the active quarter of channels is relaxed. The subsequent pointwise mixing may wash this advantage out, so we treat it as an empirical question with a pre-stated test.

Baselines

The naive reference point uses the same encoder and the same verified barrier, but the shield checks the point estimate $\hat s$ with no margin. The encoder is still trained under full randomization. This is what a practitioner would deploy by combining domain randomization with a standard barrier filter. Beating it means fewer state-wise violations under in-spec noise at comparable reward.

The fixed-buffer baseline inflates the barrier by a constant margin. The margin is tuned by grid search on the same randomized validation episodes, with the same number of tuning trials we give our method's regularizer weight. We beat it if we reach equal or lower violations with higher reward, which is evidence that an input-adaptive margin is less conservative than a global one.

The strongest statistical competitor uses a conformal margin. Split conformal prediction on randomized simulation frames, with normalized (input-dependent) scores, produces a calibrated error radius, following the methodology of Verification of neural reachable tubes via scenario optimization and conformal prediction. It uses the same calibration set size as our $e_0$. It matches us on-distribution by design. We beat it if we show lower violation rates when the in-spec noise distribution shifts within $\Theta$, for example toward its edges, at a margin no more than modestly larger. A conformal radius has no reason to hold under such a shift, whereas the certified term does.

The strongest existing pixel-based safe RL method is the latent-barrier approach of State-wise safe reinforcement learning with pixel observations. We rerun it with its published architecture under our randomization distribution and episode budget. The file reports only that it "maintained near-zero costs while matching the reward of the best baseline" on CarGoal1; no specific numerical cost or reward values were found on Lacuna for its Safety Gym sub-tasks. We will therefore reproduce its numbers ourselves before comparing.

On the LiRPA navigation setting, the strongest system baseline is Verification-Guided Shielding for Deep Reinforcement Learning. It reports a 1.5× execution overhead with the shield active 1.3% to 3.4% of the time in Mapless Navigation, under clean sensing. We beat it if our overhead stays in that range while violations stay near zero under in-spec LiDAR noise, where we expect its clean-input shield to fail.

The key ablation swaps the encoder architecture: partial convolution versus depthwise-separable versus plain, all at matched clean state-estimation error. It isolates whether partial convolution contributes anything to bound tightness or to bound-computation latency.

Why not just…?

Why not just use a conformal margin and skip formal bounds entirely? This is the most serious threat. On the calibration distribution, a normalized conformal radius may be about as tight as ours and much cheaper. Its coverage, however, is marginal over the calibration distribution. Domain randomization samples $\theta$ from a chosen prior, while reality may sit at a corner of $\Theta$. Our certified term holds for every $\theta$ in the box. The edge-shift experiment below settles the question. If the conformal margin's violation rate does not rise at the box edges, the formal term is not earning its cost, and we will say so and recommend the conformal shield.

Why not just train a noise-robust encoder and trust its output? Training-time noise injection lowers average error but gives no per-frame guarantee. Verification-Guided Shielding for Deep Reinforcement Learning found inputs causing certain collisions in every one of over 250 agents that empirically succeeded above 95% of the time. The no-margin baseline tests this directly under in-spec noise.

Why not just use a standard small CNN with interval bounds, without partial convolution? This may well work equally well. The pointwise layers that follow each partial convolution mix all channels and could erase the exact-bypass advantage. The architecture ablation decides it. If partial convolution gives neither tighter $r(x)$ nor faster bound computation at matched accuracy, we drop it, and the proposal reduces to its architecture-agnostic core.

Why not certify the full map from image to barrier value end to end? Exact verification already strains at small sizes. dReal and Z3 timed out after three hours on 64-neuron networks on a 2D system, per Exact Verification of ReLU Neural Control Barrier Functions. Verifying a convolutional encoder jointly with dynamics is far beyond that. Our split keeps the exact verification on the low-dimensional barrier and confines the image network to cheap, local, per-frame bound propagation.

Proposed Experiments

The main testbed is Safety Gym SG6 PointGoal and CarGoal from 3×64×64 pixels, as used by State-wise safe reinforcement learning with pixel observations. We add photometric, additive-noise, translation, blur and dropout randomization, plus friction and mass randomization. The second testbed is the LiDAR Mapless Navigation task of Verification-Guided Shielding for Deep Reinforcement Learning, with randomized range noise.

The metrics are:

  • state-wise violation rate per episode;
  • task reward;
  • shield activation rate;
  • per-step shield latency against a fixed control period;
  • coverage, meaning the fraction of steps where the true state lies inside $\hat s \pm (e_0 + r(x))$;
  • margin tightness, meaning certified $r(x)$ divided by the worst error found by dense sampling over $\Theta$.

All methods share encoder training data, environment steps, and hyperparameter search budget.

Three checks decide the proposal.

The first is coverage under edge shift. We test with $\theta$ drawn only from the outer 10% of each range of $\Theta$. The certified shield should keep coverage near its calibrated level, and the conformal baseline should not. If both hold, the formal term is redundant.

The second is the conservatism cost. With the bound-tightening regularizer, reward should come within a small gap of the no-margin baseline. If $r(x)$ is so loose that reward collapses, the approach is impractical at this resolution, and we will report the largest image size and network width at which it remains useful.

The third is noise outside the spec. Unseen corruption types, such as lens flare not modeled in $\Theta$, should break the formal term. We report how far violations rise, and whether a large $r(x)$ or an atypical $e_0$ score works as a cheap warning that the input is outside the spec.

The partial-convolution ablation is reported with a pre-stated criterion. Partial convolution counts as a contribution only if it improves either margin tightness or bound latency by a clear margin at matched clean accuracy.

If simulation results hold, a small zero-shot camera-robot trial would test whether real coverage tracks simulated coverage. We treat this as a stretch goal and make no performance claim in advance.