Masks That Survive Noisy Sensors: Receding-Horizon Action Masking Anchored on a Backup Plan

Action masking means removing unsafe or impossible actions from a reinforcement learning (RL) agent's choices before it picks one, instead of punishing it afterward. It makes learning fast and keeps the agent safe during training. Every existing continuous-action mask, however, is computed from the robot's state as if that state were measured exactly, and it only checks one step ahead. On real hardware both assumptions fail together. A noisy position reading moves the mask around from step to step. The mask then flickers, sometimes letting in actions that are dangerous for the true state and sometimes shrinking to nothing. A one-step check also cannot see that a fast, heavy quadrotor is already committed to a collision several steps from now.

We propose computing the mask from a short look-ahead plan that is re-solved at every step. This is receding horizon control, the idea behind model predictive control (MPC): plan N steps, execute only the first, then replan. The mask is built around a safe "backup plan" carried over from the previous step and is checked against every state consistent with the noisy measurement, not only the single estimate. The central claim is that anchoring the mask on the shifted backup plan makes it provably never empty and never unsafe under bounded measurement noise, while keeping the learning speed of pre-hoc masking. One-step masks computed from noisy estimates lack the first property. Post-hoc safety filters lack the second.

Core Claim and What Is New

The closest work is continuous action masking by Stolz et al.. They map a policy's sample from a unit box into a state-dependent zonotope of relevant actions, $c(s) + G(s)a^l$. A zonotope is a convex polytope written as a center plus a linear image of a box. They show that standard PPO fails on a 3D quadrotor where masked PPO succeeds. Their zonotope guarantees only one-step safety and assumes $s$ is known exactly. Model-free Safe Control for Zero-Violation RL shares both limits: its safe-boundary projection is one-step and uses ground-truth relative positions.

On the other side, the robust predictive safety filter of Shaw-Cortez et al. does look N steps ahead and tolerates bounded disturbances. It does so as an override applied after the policy acts, and it assumes the current state is known. Predictive Control Barrier Functions shows that a softened MPC's value function acts as a barrier function. A barrier function here means a quantity that certifies the system can stay in the safe set. That work is purely classical, has no learning, and models no sensor noise.

The new object in this proposal is an N-step, estimation-aware generator mask. Its center is the first action of the previous step's backup plan, shifted forward by one step. Its shape is the largest zonotope around that center for which every first action still admits a safe N-step continuation from every state in the current estimation set. The property we target, which none of these papers states or can state, is recursive non-emptiness under measurement noise. If the true state lies inside the estimation set at every step, the shifted backup action is always admissible, so the mask always contains at least that action. Every action the mask admits is safe for the true state.

Formally, let $\mathcal{E}_t$ be a set of states consistent with the measurement history at time $t$, produced by a set-membership or confidence-bounded filter. Let $\mathcal{X}$ be the safe state set and $\mathcal{X}_f$ a terminal invariant set. The admissible first-action set is

$$\mathcal{A}^N_t = \{u : \forall x \in \mathcal{E}_t,\ \exists\, u_{1:N-1} \text{ such that } x_{k|t} \in \mathcal{X},\ x_{N|t} \in \mathcal{X}_f \text{ with } u_{0}=u\}.$$

This set is generally non-convex, so we do not mask with it directly. Instead we inner-approximate it by a zonotope

$$Z_t = \bar u_t + G_t a^l, \quad a^l \in [-1,1]^m,$$

where $\bar u_t$ is the first action of the shifted backup plan. $G_t$ is chosen by a small linear program that linearizes the dynamics along the backup trajectory and tightens constraints by the size of $\mathcal{E}_t$ plus a linearization-error bound.

The policy samples $a^l$, and Stolz et al.'s generator-mask policy gradient applies unchanged. After execution, the backup plan is replaced by the re-solved plan whenever the executed action was not the center. If the solve fails, the old shifted plan is kept. The mask then degenerates gracefully toward the backup action and never becomes empty.

The three directions play different roles. Receding horizon control supplies the mechanism: the look-ahead and the carried-over certificate. Action masking is the interface to learning. Sensor noise robustness is the regime that breaks existing masks and the axis on which we evaluate.

Research Program

The first piece of work is the mask construction and its guarantee. We will prove recursive non-emptiness and true-state safety for known dynamics, under bounded noise and a consistent estimation set. The proof extends the shifted-sequence argument from Shaw-Cortez et al. from additive disturbances on the dynamics to uncertainty in the initial state. Because shrinking sets with a global Lipschitz constant is the known source of over-conservatism in that line of work, our tightening uses local linearization bounds along the backup trajectory. We will measure how much admissible volume this recovers.

The second piece is a diagnostic study of how existing masks fail under noise. We will quantify, as a function of noise level and model inertia, three quantities for one-step zonotope masks fed with noisy estimates. Leakage is the fraction of masked-in actions that are unsafe for the true state. Over-pruning is the fraction of true-safe actions excluded and the rate of empty masks. Mask chatter is the step-to-step change in the mask center. The file identifies this failure mode as an open gap but reports no measurement of it. This study stands as a contribution even if the method underperforms.

The third piece is learning under the mask, with deliberately mismatched noise between training and testing as a sim-to-real proxy. Policies are trained with the noise family used to size $\mathcal{E}_t$, which is white Gaussian noise. They are tested on structured corruptions the estimation set does not cover: step biases, dropouts, and impulses. Roll-Drop is orthogonal to our method, because it perturbs rollouts rather than constraining actions. We therefore test it both as a baseline and stacked with our mask. Hardware transfer is out of scope for this note. The noise-mismatch protocol is designed so that a later quadrotor deployment would test the same hypothesis.

Baselines

The naive reference point is unmasked PPO on the noisy observations. On the 3D quadrotor, Stolz et al. report that standard PPO "failed to learn a meaningful policy" even without noise. We expect the same, and it anchors the bottom of the comparison.

The penalty-based reference is PPO-Lagrangian from the SafePO library accompanying Safety Gymnasium. On velocity tasks that paper reports a 98% reduction in violations relative to PPO "with only a 45% drop in total reward." Beating it means fewer true-state violations during training at equal or higher return.

The strongest masking competitor is the Stolz generator mask computed from a Kalman-filtered state estimate. This is the obvious engineering fix and directly isolates our horizon and anchoring ingredients. A second variant inflates that one-step mask by the noise bound.

The strongest predictive competitor is the robust predictive safety filter of Shaw-Cortez et al., wrapped around PPO. We will run it with the same horizon and estimation-set tightening as our method. The policy gradient will use the executed (filtered) action, so it is not handicapped by gradient mismatch. Its paper reports qualitatively that DPC plus the filter "maintained all constraints across all tests" under noise, and that event triggering was 66.0% faster than a standard filter. We found no reported return or violation numbers on Lacuna for either Stolz masks or this filter on noisy Safe-Control-Gym quadrotors, so all such numbers will be produced here.

ISSA reports zero training violations and 95% ± 9% of baseline reward with clean states. We include it on the Safety-Gymnasium point task as a one-step projection reference.

All methods receive the same environment steps, the same network sizes, and the same hyperparameter search budget of a fixed number of configurations per method. They are evaluated on the same seeds and noise draws. Beating the field means two things on the high-inertia tasks under mismatched noise. First, strictly fewer true-state violations than the filtered one-step mask. Second, higher return or faster convergence than the safety filter at a comparable violation rate.

Three ablations isolate the key ingredient: horizon N = 1 versus larger N, the mask centered on a freshly re-solved plan rather than the shifted backup, and removing the estimation-set tightening.

Why not just…?

Why not just run Stolz's mask on a Kalman-filtered state? Filtering shrinks noise but adds lag, and a one-step mask still cannot see momentum-driven violations several steps out. That limitation is exactly why Predictive Control Barrier Functions and the N-step filter literature exist. The diagnostic study measures leakage for this baseline directly. If its leakage is near zero on the 2D and 3D quadrotors, the horizon component is unnecessary for these systems, and we will report that.

Why not just shrink the one-step mask by a noise margin? Robust shrinkage is safe when the margin is large enough. On high-inertia systems, a margin large enough to cover multi-step effects tends to empty the set. That is the same over-conservatism attributed to Lipschitz tightening in Shaw-Cortez et al.. Our empty-mask rate metric settles this. The backup-plan anchor is the specific feature that prevents emptiness, and the "fresh re-solve center" ablation shows whether it matters.

Why not just use a post-hoc predictive safety filter and train on executed actions? This is the real threat. Their filter already handles N-step safety under disturbances, and training on executed actions removes much of the gradient mismatch Müller describes. If our mask does not beat it on sample efficiency or return at matched violation rates, the learning-side claim fails. The contribution would then reduce to the noise-aware guarantee and the diagnostic study. We additionally measure intervention rate and actuator chatter: a filter that overrides often under noise produces jerky executed actions, while a mask keeps the policy's own distribution inside a smoothly moving set.

Why not just inject observation noise or use Roll-Drop? These methods make policies tolerate noise, but they give no constraint guarantee and do nothing about violations during training. Roll-Drop reports 80% locomotion success under 25% observation noise, versus below 40% for baselines, with no safety constraints in play. We test it both alone and combined with our mask, because these approaches are complementary rather than exclusive.

Proposed Experiments and Decisive Checks

The main testbed is the 2D quadrotor and cart-pole in Safe-Control-Gym. It provides CasADi symbolic models for the mask's linear programs and built-in white-noise, step, and impulse injection on observations. We add the Seeker reach-avoid and 3D quadrotor tasks from Stolz et al., and the Safety-Gymnasium point-goal task with noisy lidar.

Metrics are computed against the true simulator state, not the observation. They are cumulative violations during training and at test, return, steps to a fixed return threshold, empty-mask rate, intervention rate for filters, action total variation, and per-step wall-clock time. Wall-clock time matters because the MCMC mask was up to 170 times slower than the baseline; ours must stay within real-time rates.

Three checks are decisive. First, under in-distribution bounded noise, our mask must show zero true-state violations and zero empty masks during training, while the filtered one-step mask shows nonzero leakage on the high-inertia tasks. If not, the theory or its implementation is wrong. Second, under mismatched structured noise, we report how violation counts degrade once the consistency assumption breaks. The honest prediction is graceful degradation toward the backup action, not zero violations. Third, against the predictive safety filter, the method should reach the return threshold in fewer steps with lower chatter at matched violation rates. Failing that check is the outcome that would most change what this work claims.