When a robot arm reaches in to push or grasp something, its own links block the camera. It goes blind at the moment it is changing the scene, and at the moment it most needs to check that it is not about to knock something over. Safe exploration methods that learn from images mostly ignore this. They check whether a planned motion ends somewhere the robot "knows how to get back from." They make that check on an image encoding that was never trained to tell "I can see nothing is wrong" apart from "I cannot see." We propose to make visibility part of what it means for a state to be safe. The robot may enter a configuration where its arm hides the task region, but only if every plan it commits to ends in a state that is both known to be recoverable and visible again. We call this a return-to-sight constraint.
The mechanism depends on a simple observation. Self-occlusion is not random noise. Given the robot's joint angles, its mesh, and a calibrated camera, we can compute exactly which pixels the arm will cover at any future step of a plan. We use this predictable mask twice. First, we train the image model cheaply: we paste rendered arm masks over images where the scene was visible, so the model learns to report high uncertainty when it is blind. The robot never has to experience occluded failures to learn this. Second, the safety check uses those masks: occluded steps are treated as uncertain, and the plan must end where the camera can confirm the outcome. The goal is safe exploration for image-based manipulation with fewer violations during training and no more samples than an occlusion-blind learner.
Core Claim
We build on LS3. LS3 encodes camera images into a 32-dimensional code with a variational autoencoder, a network that compresses an image into a short vector and reconstructs it. It then learns a classifier over that code that predicts whether task completion is still reachable, trained recursively from successful trajectories. Finally, it plans with the cross-entropy method, a sampling-based optimizer, under the rule that each plan's final state must lie in this learned safe set.
LS3 treats the image as a faithful reading of the state. We claim this assumption fails under self-occlusion in a specific and dangerous way. An occluded image maps to a confident code that looks like a familiar safe state, so the safe-set classifier says "safe" exactly when it has no evidence. Our contribution is to replace membership in the safe set with a stricter test at the end of each plan:
$$\Pr\big[z_H \in \mathcal{S}\big] \ge 1-\delta \quad\text{and}\quad u(z_H) \le \tau \quad\text{and}\quad \textstyle\sum_{t\le H} m_t \le K.$$
The symbols are as follows. $z_H$ is the predicted latent state at the end of the plan horizon $H$, and $\mathcal{S}$ is the learned safe set. $u(\cdot)$ is disagreement across an ensemble of encoder–dynamics models, which is calibrated to rise under occlusion. $m_t \in \{0,1\}$ marks whether the rendered arm mask covers more than a set fraction of the task region at step $t$. $K$ caps how many consecutive steps the robot may spend blind.
The same probability threshold also applies to every intermediate occluded step, computed over the ensemble's spread rather than a single rollout. That borrows the worst-case-over-hidden-region logic that OA-MPC uses for obstacles emerging from blind corners in mobile navigation. OA-MPC assumes a known speed bound on hidden agents. Our hidden agent is the scene the robot itself is pushing, and our "speed bound" comes from the learned dynamics ensemble instead.
Research Program
We would build three pieces on top of LS3's code. The first is a mask renderer. It takes planned joint trajectories and produces per-step pixel masks of the arm, using the simulator's robot model or the calibrated real robot. The second is an occlusion-augmented training procedure for the encoder and latent dynamics ensemble. We overlay rendered masks on visible frames and train each ensemble member on different mask samples, so that disagreement tracks how much of the task region is hidden. The third is the return-to-sight planner, which is LS3's planner with the terminal test above.
The analysis question is whether the augmentation actually produces calibrated blindness. We test this by holding out real occluded frames. If ensemble disagreement on those frames does not predict latent prediction error, the constraint has no teeth. We would then fall back to an explicit visibility classifier on the rendered mask, and report that as a finding.
The work is motivated by two observations. RAIL uses polynomial zonotopes, a set representation that over-approximates the volume the arm sweeps through, to filter imitation policies. It names the "observation bottleneck" of assuming known obstacle geometry as its main open problem. ContactGuard shows that a latent world model can forecast pre-contact failures under wrist-camera occlusion, but it can only abort.
The work is also constrained by the "safety paradox" of Feasible Dual Policy Iteration: a safer policy collects fewer boundary examples. This is the reason we synthesize occluded training data from masks rather than waiting to encounter occluded failures. RESPO must experience violations to learn its reachability estimate, which is the cost we avoid.
The idea is not simply to "add an occlusion penalty to LS3." The recoverability label is unchanged. What changes is when the planner is allowed to trust it.
Baselines
The naive reference point is LS3 run unchanged on occlusion-heavy variants of its own tasks. This measures how much self-occlusion hurts an occlusion-blind safe-set planner. The strongest reported number on the base task is LS3's 64.8% success on simulated Sequential Pushing, against 42.5% for SACfD and 0.6% for AWAC, as reported by LS3. No reported result was found on Lacuna for LS3 or any competitor on an occluded variant of these tasks, so we will establish that number ourselves.
Beating LS3 means one of two outcomes on the occluded variants. We either match its success rate with significantly fewer cumulative constraint violations during training, or we reach higher success at the same violation count. All runs use identical demonstrations, environment steps, planner sample budgets, and a matched hyperparameter sweep over LS3's own safe-set threshold.
The strongest existing filter-style competitor is a RAIL-style swept-volume filter. It is built on a last-seen occupancy map of the objects in the scene, which is our fairest adaptation of RAIL to a setting without ground-truth geometry. RAIL reports a 0% collision rate and raises Diffusion Policy's safe success in PointMaze from 15% to 97% when geometry is known. We will wrap the same learned policy with this filter and give it the same perception inputs.
We include a Lagrangian constrained baseline, PPO-Lagrangian from the SafePO library in Safety Gymnasium, with the constraint cost augmented by an occlusion term. We also include an abort-only baseline: a ContactGuard-style latent failure classifier that freezes the arm and requests a reset.
Two ablations isolate the key ingredients. In the first, mask augmentation trains the ensemble but the terminal visibility test is removed. This asks whether better representations alone suffice. In the second, the visibility test uses mask coverage alone, without augmentation-calibrated uncertainty. This asks whether a purely geometric rule suffices.
Why not just…?
Why not just remember the last-seen scene and check the plan against that? This is the real threat. For static obstacles it probably works as well as our method, because nothing changes while the robot is blind. Our claim rests on the fact that in pushing and contact tasks, the hidden objects are the ones the robot is moving, so the memory goes stale exactly during contact. We include the last-seen-memory RAIL baseline and split results into static-obstacle and pushed-object conditions. If memory matches us on pushed-object tasks, the return-to-sight constraint is unnecessary and we will say so.
Why not just add more fixed cameras? AV-ALOHA reports that static camera setups reach only 30% on Thread Needle versus 52% with an actively moved camera, and 40% versus 60% on Hidden Pick. So fixed views leave real blind spots in tight workspaces. Its 21-dimensional action space, however, shows the sample cost of fixing occlusion with more hardware degrees of freedom. We will also run LS3 and our method with a second fixed camera. If a second camera closes the gap, our method's value is limited to single-camera or wrist-camera deployments, and we will report it that way.
Why not just penalize occlusion in a constrained RL objective? Safety Gymnasium reports that PPO-Lagrangian cuts violations by 98% relative to PPO but loses 45% of reward. It also guarantees constraints only at convergence, so violations during training remain. The PPO-Lagrangian baseline, measured by cumulative violations over training rather than final cost, tests this directly.
Why not just abort when a failure is predicted? ContactGuard predicts pre-contact failure with ROC AUC 0.992 on its cup task. However, it offers no way back, so every abort becomes a manual reset, which is the bottleneck Leave no Trace addressed by learning reset policies that cut Pusher resets 2.5×. Our comparison counts manual resets for the abort baseline against our planner.
Proposed Experiments
The main testbed is the LS3 suite: Reacher and Sequential Pushing. We re-place the camera so the pusher crosses the line of sight to the blocks during contact. The constraint is an unsafe event: a block leaving the workspace, or contact with a designated fragile region.
The second testbed is a set of RLBench tasks restricted to a single over-the-shoulder camera, under the benchmark's sparse reward. This adds realism and variety of arm occlusion, with a collision-with-distractor cost. RVT reports 62.9% average success on 18 RLBench tasks, but that is with multi-view inputs and imitation only. It serves as a context point, not a comparable baseline.
The metrics are success rate against environment steps, cumulative constraint violations during training, manual resets, and the fraction of time spent occluded. We also measure calibration: we compute the correlation between ensemble disagreement and actual latent prediction error on real occluded frames that were held out from training.
Three decisive checks follow. First, on pushed-object occluded tasks, our method must reduce training-time violations relative to both unchanged LS3 and the last-seen-memory filter, at no more than a small loss in steps-to-success. Second, the augmentation-only ablation must fall short of the full method. Otherwise the planning constraint adds nothing. Third, disagreement must be calibrated on real occlusions. Otherwise the synthetic masks do not transfer, and the premise that self-occlusion can be rendered instead of experienced is wrong.
References
- LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of Sparse Reward Iterative Tasks
- OA-MPC: Occlusion-Aware MPC for Guaranteed Safe Robot Navigation With Unseen Dynamic Obstacles
- RAIL: Reachability-Aided Imitation Learning for Safe Policy Execution
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Iterative Reachability Estimation for Safe Reinforcement Learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
- RLBench: The Robot Learning Benchmark & Learning Environment.
- RVT: Robotic View Transformer for 3D Object Manipulation