A robot arm learning by trial and error (reinforcement learning, RL) needs two things that conflict near contact. It must not hit things it should not touch, and it must see what it is doing. Its own arm and gripper often block its cameras exactly when the stakes are highest. Existing safety layers for RL ignore this. They decide which actions are allowed from a fixed model of the world, as if the robot always knew where every obstacle was. We propose an action mask, meaning a filter applied before the policy samples, that restricts the arm to motions provably clear of obstacles given the robot's current uncertainty about where those obstacles are. That uncertainty grows whenever the robot's own body hides an obstacle from the cameras. The result is that occluding the view automatically shrinks the set of motions the robot is allowed next.
The point of this coupling is that it makes seeing useful without ever rewarding it. A policy that keeps fragile or moving objects in view keeps a large set of allowed actions. A policy that blindly sweeps its arm across the camera finds itself boxed in. We expect visibility-preserving behavior to emerge from the safety constraint alone, without a hand-tuned occlusion penalty. We also expect zero, or bounded, collisions during training, which penalty-based safe RL does not deliver.
Core Claim
A belief-dependent, uncertainty-inflated continuous action mask, combined with a simple model of how self-occlusion inflates that uncertainty, does two things on vision-based manipulation tasks with moving or perturbed obstacles. First, it keeps training-time collisions near zero. Second, it reaches a better trade-off between task success and safety than any fixed-margin mask, because the policy learns to manage what it can see.
The claim is falsifiable in a specific way. If a well-tuned constant safety margin matches our Pareto frontier of success versus collisions, the occlusion coupling adds nothing, and the idea fails.
Positioning Against Prior Work
The method comes from continuous action masking. Excluding the Irrelevant confines policy samples to a state-dependent convex set using a Ray Mask, which squashes actions radially toward the set boundary, and a Generator Mask built on zonotopes, a type of convex polytope. It reports that standard PPO "failed to learn a meaningful policy" on a 3D quadrotor where masked policies succeeded. However, its sets come from known analytic dynamics, and it was tested only on low-dimensional control.
Provably Safe Reinforcement Learning places masking alongside projection and replacement. It reports 100% safety compliance during training on a pendulum and a 2D quadrotor, and it also assumes a conformant model is known a priori.
We keep their machinery and change what the safe set depends on. Our safe set is computed from a perceptual belief whose quality the agent's own actions degrade.
Two occlusion papers motivate the work but stop short of online RL. MobileH2R penalizes self-occluding arm poses with a "Vision Neural Loss," but only inside an offline demonstration planner. ContactGuard predicts pre-contact failures from multi-view latents with high accuracy (ROC AUC 0.992 on Cup, 0.946 on Box). It can only veto and halt a fixed imitation policy, though; it cannot reshape what the policy explores.
Our proposal differs from both. Occlusion is never itself a constraint. It enters only through its effect on the margin of the collision constraint, and it acts online during exploration.
Two further papers constrain the design. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that learned feasibility estimates degrade as agents become safe and stop seeing violations. This is why we compute the mask from geometry plus a belief tracker rather than from a learned cost critic. Active Vision Might Be All You Need shows that adding viewpoint control helps on occluded tasks (52% vs 30% on Thread Needle) but hurts on simple ones through unconstrained exploration in 21 degrees of freedom. This warns us that extra freedom without structure is costly, so we do not add a camera arm in the main study.
Research Program
Belief tracker. The system has three components. The first maintains a position belief for each tracked obstacle from what the cameras actually see. Each obstacle has a mean position $\hat p_{i,t}$ and a scalar standard deviation $\sigma_{i,t}$. Measurements come only from visible pixels of segmentation and depth in a fixed shoulder camera and a wrist camera. When the obstacle is mostly hidden, the measurement noise is large or the update is skipped. Uncertainty then grows by a process-noise term $q$ that reflects how fast obstacles can move: $\sigma_{i,t+1}^2 = \sigma_{i,t}^2 + q$ when occluded. When the obstacle is well visible, the uncertainty resets toward sensor noise.
Because robot kinematics and camera extrinsics are known, the visibility of each obstacle under a candidate end-effector motion can be predicted by rendering the arm's silhouette. This makes the occlusion consequence of an action computable before it is taken.
Mask. The second component is the mask itself. Over one control step, the allowed end-effector displacements $\Delta x$ form a local convex polytope:
$$\mathcal{A}_t = \{\Delta x \in B : n_{i,t}^\top (x_t + \Delta x) \ge n_{i,t}^\top \hat p_{i,t} + r_i + \kappa\, \sigma_{i,t+1}(\Delta x)\ \ \forall i\}.$$
Here $x_t$ is the gripper position, $B$ is the controller's box of displacement limits, $n_{i,t}$ is the unit direction from obstacle $i$ toward the gripper, $r_i$ is a clearance radius, and $\kappa$ sets the confidence level. The $\sigma_{i,t+1}(\Delta x)$ term is the predicted next-step uncertainty if $\Delta x$ is executed. We bound it conservatively by its value under maximal occlusion within the step, which keeps the set convex.
This construction sidesteps the convexity limitation that Excluding the Irrelevant cannot handle. It does so by being local, not by being clever. The globally non-convex free space is inner-approximated at each step by a polytope that contains zero displacement whenever the current state is safe. We use the Ray Mask over this polytope. We avoid the Distributional Mask, which that paper reports running "up to 170 times slower" in Walker2D. The target object is excluded from the obstacle set, so contact with it remains permitted.
Policy. The third component is a PPO policy acting through operational-space control in robosuite. It observes images, proprioception, and the current $\sigma_{i,t}$ values, so it can learn how its motions affect future freedom.
Guarantee. We will state a conditional guarantee. If obstacle displacement per step is sub-Gaussian with variance at most $q$, each step's collision probability is bounded by the Gaussian tail at $\kappa$. We will test empirically what happens when obstacles move faster than $q$ assumes.
Tasks. We will build three robosuite tasks on the Panda arm, all with a shoulder camera and a wrist camera. The first is reaching a target among fragile cups that a scripted disturbance occasionally nudges. The second is picking a target whose neighbor drifts slowly. The third is placing into a bin while a scripted "hand" obstacle moves through the workspace.
All three are designed so that the direct approach path self-occludes a nearby obstacle. A good policy must either approach from a viewpoint-preserving angle or pause to re-observe.
Baselines
Naive reference. Unconstrained PPO on the same observations and controller, with task reward only. It shows how many collisions occur by default and what success is reachable with no safety mechanism.
Strongest penalty-based competitor. PPO-Lag and CPPO-PID from the SafePO library of Safety-Gymnasium, with a per-step collision cost. They are the standard model-free answer to safe exploration, and the PID variant specifically damps multiplier oscillation. The only reported figure on Lacuna for them is in a different domain: on velocity-constrained tasks, "PPO-Lag achieved a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward." We found no reported result on Lacuna for PPO-Lag on robosuite manipulation with self-occlusion. Beating it means lower cumulative training-time collisions at equal or higher final success.
Strongest masking competitor: the fixed-margin mask. The identical Ray Mask polytope, but with the uncertainty term replaced by a constant inflation $c$. We sweep $c$ over the full range from optimistic to maximally conservative, using the same number of tuning runs as our $\kappa$ sweep. This is the ablation that isolates the key ingredient. Each value of $c$ gives one point on a success-versus-collision plane. We claim a win only if our method lies strictly outside the convex hull of those points.
Action replacement. It uses the same belief-dependent safe set, but proposed actions outside it are replaced by zero motion rather than masked. Provably Safe Reinforcement Learning found that replacement "consistently outperformed more complex methods in terms of total reward and learning speed." We include the adaptation penalty that paper reports as helpful.
Hard visibility mask. This is the obvious idea suggested by MobileH2R's loss. It additionally forbids any action that drops an obstacle's visible fraction below a threshold.
Shared protocol. Every method uses the same tasks, seeds (at least five), environment-step budget, network, and hyperparameter search budget.
Why not just…?
Why not just add a collision cost and run a Lagrangian method? Lagrangian methods enforce constraints only on average and only near convergence, so they collide repeatedly while learning. Quanyi Li et al. document the fragile middle ground between ignoring penalties and freezing. Our cumulative-collision curves over training will show this directly. The threat is real if CPPO-PID converges to similar final success: then our advantage reduces to training-time safety. That would still be the main point for physical robots, and we will say so plainly.
Why not just use a fixed, conservative safety margin? This is the most serious alternative. A large enough constant margin is safe under occlusion, and a small one is efficient when everything is visible. Only the margin sweep described in the Baselines section can settle whether adapting the margin to what the robot sees beats every constant choice. If it does not, we will report that the occlusion coupling is unnecessary in these tasks.
Why not just forbid occluding motions outright? In the final approach the gripper must hide nearby geometry, as VFAS-Grasp and ContactGuard both describe. A hard visibility mask should therefore make grasps near obstacles infeasible. Our formulation allows brief occlusion when obstacles are known to be slow or recently seen. The hard-mask baseline tests this prediction.
Why not just add cameras or re-render a 3D view like RVT? RVT reaches 62.9% average success on RLBench by re-projecting point clouds, but it depends on precise multi-camera calibration and struggles with thin objects. Extra camera rigs are impractical on mobile platforms like MobileH2R's. We run a four-camera condition as an upper reference. With good coverage, uncertainty rarely grows and our mask should collapse to the fixed mask, which we expect and will confirm. The contribution is for wrist-plus-one-camera setups, where extra cameras cannot be added.
Why not just stop when a world model predicts failure, as ContactGuard does? A veto halts the task and gives the policy no gradient toward safer behavior. Stopping is also not reliably safe when obstacles move. The action-replacement baseline is the RL analogue of that design, and it isolates veto-style intervention from masking under an identical safe set.
Proposed Experiments
Training safety and the Pareto frontier. The first experiment compares training-time safety and final performance. We plot cumulative collisions with obstacles against environment steps, together with final success rate on held-out obstacle motions. From the fixed-mask sweep we construct the Pareto frontier of success versus collisions.
Emergent visibility. The second experiment tests whether visibility behavior emerges. During the last 10 cm of approach we measure each obstacle's mean visible fraction, along with how often the policy pauses to re-observe. We compare this against the fixed-mask and penalty policies. Neither our policy nor its baselines receives any visibility reward, so a significant increase for our method alone would confirm the mechanism. No increase, together with no frontier gain, would falsify it.
Robustness to model mismatch. The third experiment runs obstacle speeds at one, two, and four times the assumed rate $q$. We report how gracefully violation rates degrade, honestly testing the dependence on a conformant model that Provably Safe Reinforcement Learning flags.
Compute cost. We report mask computation time per step against the policy step time, to show that the local polytope and Ray Mask stay within real-time budgets.
References
- Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking
- Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking
- MobileH2R: Learning Generalizable Human to Mobile Robot Handover Exclusively from Scalable and Diverse Synthetic Data
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- VFAS-Grasp: Closed Loop Grasping with Visual Feedback and Adaptive Sampling
- RVT: Robotic View Transformer for 3D Object Manipulation