When a robot arm reaches for an object, its own gripper eventually blocks the camera's view of that object, so the last and most delicate centimeters of a grasp are often executed without sight. This note proposes treating the configurations where the camera can still see the contact region as the robot's certified safe starting set, the "safe seed" that safe-exploration methods need before they can learn anything. Pulling back to the last configuration where the target was fully visible serves as a backup action that is safe by construction. The robot is then allowed a small, per-region number of "blind steps" past the point where sight is lost. That allowance grows on the real robot only where a simulation-trained predictor has been shown to guess correctly what the robot will see when the object comes back into view.
The two-sentence version is this. Safe exploration needs a known-safe starting point, and in vision-based manipulation the natural one is "configurations where I can see what I'm about to touch, plus a retreat back to them." A predictor trained in randomized simulation decides how far past that point the robot may go blind. The real robot widens that distance only where its own re-observations confirm the predictor is right. What is new is the expanding quantity. Earlier methods grow a set of safe states or mix in a safe policy. Here the thing that grows is a budget of blind motion, and it is checked with labels the robot collects for free by looking again, not with constraint violations.
Why this gap, and what already exists
Safe exploration theory is blunt about seeds. Without prior knowledge of catastrophic actions, safe learning with vanishing regret is impossible. That is why SafeOpt-style Gaussian-process expansion, GOOSE, and ActSafe all start from a known safe set or safe policy and grow it outward. ActSafe is the closest method template. It grows the safe set using an ensemble of learned dynamics models, optimistic about dynamics uncertainty and pessimistic about predicted costs. It runs only in simulation, though, and it needs its seed handed to it. The remaining question is where a seed comes from on a real manipulator whose sensor fails exactly at contact.
Two papers connect safe exploration to sim-to-real directly, and both constrain the design. Provably Safe Sim-to-Real Transfer mixes an exploratory policy with a known safe policy. It uses a linear program to set the mixing weight, and it updates the dynamics model only where simulator and reality disagree. It reports 0% unsafe iterations against 68.5% for unconstrained transfer, but only on a fully observed 5x5 gridworld. We keep its core logic: explore only as much as a safe fallback can cover, and spend real samples only on the mismatch region. We replace probabilistic mixing with gating, because a continuous arm cannot "mix" a grasp.
Sim-to-Lab-to-Real trains a reachability-based safety critic from images and uses it to override a performance policy during lab fine-tuning. It reports 76.7% success and 86.7% safety on a quadruped. Its critic assumes the camera sees the obstacles, and in manipulation the robot's own body removes that assumption.
The occlusion literature supplies the sensing side. ContactGuard rolls a learned latent world model forward through the wrist camera's terminal blind zone and aborts grasps predicted to fail. It reports ROC AUC 0.992 on cup pick-and-place and 0.946 on box pick-and-place. However, it only aborts, it trains on real rollouts, and it uses about 250 labeled attempts per task. MobileH2R shows that planning demonstrations to avoid self-occlusion also cuts human-robot collisions by about one third. That is evidence that visibility and safety are coupled, but the paper uses a fixed planning penalty rather than a learned, adaptive limit.
On the transfer side, Asymmetric Actor Critic provides the training trick we reuse. The value function sees true simulator state while the policy sees only randomized images, and the resulting policy transferred zero-shot in 5/5 real Fetch trials. That paper has no cost function at all.
A tension frames the labeling design. Breaking Safety Paradox and Blind Spot Detection both find that safe or optimal behavior starves a safety model of boundary data. The latter reports blind-spot AUC 0.96 from suboptimal data against 0.60 from optimal data. We sidestep the problem rather than court violations. Our calibration labels come from re-observing the scene after blind motion, which happens on every episode whether or not anything goes wrong.
Core Claim
In vision-based manipulation, the certified safe seed should be the visible set together with a retreat policy. The visible set $V$ is the configurations where the target's contact region is seen by at least one camera. The retreat policy reverses the just-executed free-space path back into $V$. The quantity that safe exploration expands is a per-region blind-step budget $k_b$, where $b$ indexes coarse bins of approach phase and gripper-to-object pose.
At time $t$, with latent state $z_t$ from the image history and a proposed action chunk $a_{t:t+k}$, the robot may continue blind for $k \le k_b$ steps only if the pessimistic ensemble estimate of cost, including the cost of retreating afterward, stays under budget:
$$\max_{m=1..M}\Big[\hat C_m(z_t, a_{t:t+k}) + \hat C_m^{\mathrm{ret}}(z_{t+k})\Big] \le \epsilon .$$
Here $\hat C_m$ is ensemble member $m$'s predicted cost over the chunk, $\hat C_m^{\mathrm{ret}}$ is its predicted cost of retreating to $V$, and $\epsilon$ is the per-episode safety budget. The costs are object tipping, contact force above a threshold, and arm-environment collision.
After each real episode, the robot measures a calibration error $e_b$ in each bin it entered blind. This is the gap between the forecast object pose and the pose observed once the object is visible again, either after retreat or after lifting. The budget is updated as $k_b \leftarrow k_b + 1$ if $e_b \le \delta$ over the last $n$ visits, and as $k_b \leftarrow \max(0, k_b - 1)$ otherwise.
Starting from $k_b = 0$ reproduces "never move blind," which is the conservative seed. A fixed large $k_b$ reproduces an uncertified blind policy.
Research Program
The simulation stage uses robosuite with a Franka Panda under operational-space control, on Lift, Pick-and-Place, Nut Assembly, and a fragile-object variant that adds a tip-over cost. We apply standard visual randomization in the style of (CAD)²RL and dynamics randomization as in Peng et al.. We add one targeted ingredient, occlusion randomization. Camera poses, gripper finger geometry, and object size are sampled with extra weight on configurations where the renderer's segmentation shows the target becoming invisible before contact. This follows the logic of boundary sampling: the randomization budget is spent near where visibility is lost.
We train three components in simulation. The first is a task policy that sees only images, trained with a privileged-state critic and a PID-Lagrangian cost constraint as in Safety-Gymnasium. The second is an ensemble of action-conditioned latent forecasters. Each predicts object pose, cost, and retreat cost over short action chunks, supervised by privileged simulator state. The third is a visibility classifier trained from renderer segmentation.
The real-robot stage uses a Franka Panda with one fixed camera and one wrist camera. Budgets start at zero everywhere and expand under the rule above, with no human labels.
The analysis asks three questions. First, does the predictor's calibration on real re-observations correlate with blind-motion safety? Second, how does the budget grow per bin over real episodes? Third, where does it stay at zero? We expect it to stay low for small objects inside the fingers, which would indicate where the approach reaches its limits.
Baselines
The naive reference is the same randomized, cost-constrained image policy deployed zero-shot with no gating. It carries the full sim-to-real pipeline but no seed. Beating it means strictly fewer real violations at equal or higher success. No reported result found on Lacuna for cost-constrained robosuite transfer, so this baseline is established by our own runs.
The strongest existing competitor for the occlusion part of the problem is a ContactGuard-style abort monitor (ContactGuard). It uses the same latent ensemble and the same simulation training, but it only halts and retreats when predicted failure exceeds a threshold. It never widens its blind allowance. ContactGuard's reported ROC AUC of 0.992 on cup pick-and-place came from real training data. Our version is sim-trained, so its real-world AUC is itself a measurement. It gets the same forecaster, the same real-episode count, and its threshold tuned on the same held-out simulation split. Beating it means higher success at no more violations, because an abort-only system gives up grasps that the expanded budget completes.
The strongest safe sim-to-real competitor is a Sim-to-Lab-to-Real style shield (Sim-to-Lab-to-Real). This is a reachability safety critic trained from images that overrides the policy with a backup whenever the predicted margin is breached, run with an identical real-episode budget. Its reported 86.7% safety rate is on quadruped navigation, not manipulation, so it is context rather than a target. We expect its symmetric image critic to degrade exactly where the target is occluded.
The ablation that isolates the key ingredient is a sweep over fixed blind budgets $k \in \{0, 2, 4, 8, \infty\}$ tuned in simulation, with everything else unchanged. Two further ablations remove occlusion randomization and replace re-observation labels with violation-only labels.
Why not just…?
Why not just randomize more broadly and deploy zero-shot? Randomizing textures cannot reveal an object hidden behind the fingers, which is the representational blind spot documented by Blind Spot Detection. Unconstrained transfer in the one paper that measures it produced 68.5% unsafe iterations (Provably Safe Sim-to-Real Transfer). This is only gridworld evidence, so the naive baseline in our experiments is the real test.
Why not just abort whenever vision is lost or a monitor fires? This is the most serious competitor, because it is simple and safe. It trades success for safety, and on small objects, where occlusion before contact is unavoidable, it would never finish the grasp. The abort-only baseline measures that trade directly. If it matches our success rate, the expansion mechanism is unnecessary and the contribution shrinks to sim-trained monitoring.
Why not just pick one good blind-step budget in simulation? This is a real threat. If the reality gap under occlusion is uniform across approach regions, a single tuned $k$ could match adaptive per-region budgets. The fixed-$k$ sweep settles it. The adaptive method must dominate the whole sweep's success-versus-violation curve, not just one point on it, or we would conclude that per-region certification is not needed.
Why not just add cameras? Active observer arms (Active Vision) reduce wide-angle occlusion, but they cannot see into the gap between fingertips and a small object. They also add hardware and a second collision risk. We include a two-fixed-camera condition to measure how much of the blind interval remains.
Proposed Experiments
First, a proxy-reality study in simulation gives enough trials for violation statistics. The "real" environment uses contact parameters outside the randomization range, a different renderer, and unseen camera poses. For each method we count violations over the first 200 exploration episodes and track success rate against episodes.
Second, a Franka hardware study uses the fragile-object and Pick-and-Place tasks with 100 exploration episodes per method. It reports success, violations, time spent blind, and the correlation between per-bin calibration error $e_b$ and observed failures.
The decisive checks are three. The first is whether adaptive budgets beat the entire fixed-$k$ curve. The second is whether they beat abort-only on success at equal violations. The third is whether calibration measured by re-observation predicts real blind-motion failures better than ensemble disagreement alone, since that is the link that makes the budget update safe.
A negative result on the first check would redirect the work toward sim-trained monitors. A negative result on the third would mean re-observation is not a trustworthy certificate, and the claim would fail.
References
- Safe Exploration for Optimization with Gaussian Processes
- Safe Exploration for Interactive Machine Learning
- ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning
- Provably Safe Sim-to-Real Transfer
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- MobileH2R: Learning Generalizable Human to Mobile Robot Handover Exclusively from Scalable and Diverse Synthetic Data
- Asymmetric Actor Critic for Image-Based Robot Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Blind Spot Detection for Safe Sim-to-Real Transfer
- robosuite: A Modular Simulation Framework and Benchmark for Robot Learning
- CAD2RL: Real Single-Image Flight Without a Single Real Image
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
- Efficient safe learning for controller tuning with experimental validation
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation