When a robot arm reaches for an object, the straight-line approach is usually the shortest one, and it is often the worst. As the gripper closes in, the arm, wrist and fingers block the cameras' view of exactly the region that needs millimeter corrections. The final centimeters are then executed nearly blind. Two configurations a few centimeters apart in space can differ greatly in how likely the robot is to finish the task: one leaves the target visible, the other does not. A robot that judges progress by straight-line distance will steer itself into the blind configuration.
This proposal treats self-occlusion as a case of a known failure mode in learning-based control, the Euclidean distance bias: mistaking geometric closeness for real closeness to the goal. We will learn, from camera observations, an asymmetric distance (a quasimetric, meaning the cost from A to B need not equal the cost from B to A) that measures how many closed-loop steps separate the current view from task success. Under this distance, views where the target is hidden sit far from success even when the gripper is geometrically close. When a failure is forecast before contact, the robot uses the distance to pick a recovery waypoint, a short retreat to a pose that can actually see the target, instead of only stopping.
Core Claim
The gap between geometric and true distance has been studied almost entirely as a property of physical states: walls, obstacles and kinodynamic limits that make nearby coordinates far apart. Graph-Based Hindsight Goal Generation (G-HGG) replaces Euclidean goal distance with shortest paths on an obstacle-aware grid. ARC (Learning Actionable Representations with Goal-Conditioned Policies) warps a latent space so that states behind barriers are far apart, measuring distance by how differently a goal-reaching policy acts toward them. Our claim is that under manipulator self-occlusion the gap depends on what the robot sees, not only on where it is. The same physical gripper pose is near success if the target is visible and far from it if the target is hidden.
A distance that ignores the observation cannot represent this. That holds whether it is computed from coordinates or from a graph over coordinates. A distance learned over observations from closed-loop success data can. Two properties make it useful for recovery: it must be directed, and it must compose. It is directed because backing away from a blind pose is cheap, while recovering after the object has been nudged is not. It composes through the triangle inequality, so that "here to waypoint" plus "waypoint to success" is a meaningful planning cost.
The closest existing work bounds this claim from three sides. METRICMM (Geometry of Uncertainty) motivates the proposal: it learns a latent space whose Euclidean distance tracks transition counts and uses it to down-weight occluded sensors. Its occlusions are synthetic patches and sensor drops rather than the arm's own motion, its metric is symmetric (its authors flag quasimetrics as needed), and it only filters; it never changes the trajectory. ContactGuard (Pre-Contact Execution Monitoring) motivates the recovery half. Its action-conditioned latent world model forecasts pre-contact failure with high accuracy, but its own limitation statement is that it only knows how to stop. MobileH2R (Human to Mobile Robot Handover) constrains the design. It shows that a hand-engineered visibility loss in the demonstration planner already helps. We therefore need to show what a learned, success-grounded distance adds beyond explicit line-of-sight geometry.
Research Program
The work has three parts: building the distance, checking that it captures occlusion, and using it to recover.
The first part is data. In RLBench (RLBench), whose Franka arm carries over-the-shoulder, front and eye-in-hand cameras, we select tasks whose final approach is occlusion-prone, such as grasps from above, insertions and placing into containers. We roll out a base imitation policy, either RVT (Robotic View Transformer) or a diffusion policy. We randomize the approach direction and the pre-grasp offsets so that visibility of the target varies widely at matched geometric distances. The simulator also lets us record ground-truth target visibility per camera, which we never give to the learned model but use for analysis. Each trajectory provides observation histories, end-effector poses, and the number of steps until success. Failed trajectories are marked as never reaching success.
The second part is the distance itself. A pretrained visual encoder, frozen, is followed by a small head that maps a short history of multi-view images and an optional goal image to an embedding. The distance between embeddings is computed by a parametrization that guarantees non-negativity and the asymmetric triangle inequality. We train it so that the distance from an observation to a later observation in the same successful trajectory matches the logged step count. The distance to success from observations on failed trajectories is pushed toward a cap. This is a temporal-distance objective of the kind METRICMM uses, with two changes: the symmetric constraint is dropped, and failure data enters training. The failure data is what lets "blind" observations become far.
Proprioception is deliberately excluded from the encoder at first. ContactGuard reports that adding joint states made its monitor latch onto proprioceptive shortcuts. Reintroducing proprioception through a separate, dropout-regularized channel is an explicit ablation.
The third part uses the distance for recovery. Before contact, we compute the distance from the current observation to success along the base policy's planned action chunk. When that distance exceeds a threshold, we treat it as a forecast failure. The distance is computed either directly or on latents rolled forward by a ContactGuard-style predictor. Instead of aborting, we score candidate waypoints. These are stored observations from successful dataset trajectories in the same scene configuration, together with their end-effector poses. Each candidate is scored by the composed cost of the distance from now to the waypoint plus the distance from the waypoint to success. The chosen waypoint lies in free space before contact, so we move to its stored pose with a standard motion planner and then hand control back to the base policy.
The method therefore needs no new action space. That matters because AV-ALOHA (Active Vision Might Be All You Need) reports that adding a camera arm sometimes hurt simpler tasks by inflating the action space to 21 degrees of freedom.
Baselines
The naive reference point is the base policy with no monitor. In a second variant, the base policy is wrapped in a retry rule: on any failed grasp, retreat a fixed height and re-run the policy. Both use the same demonstrations and the same evaluation seeds. Beating them shows that recovery helps at all.
The strongest existing policy on this benchmark is RVT, which re-renders the multi-view point cloud into virtual orthographic views that sidestep physical camera occlusion. RVT reports a 62.9% average success rate across 18 RLBench tasks against 49.4% for PerAct (RVT). RVT's authors note it struggles with thin objects and depends on precise calibration. On our occlusion-prone subset no reported result was found on Lacuna, so we will rerun RVT ourselves on that subset with its published training recipe. We will report it both alone and as the base policy under our recovery layer. Beating it means a statistically clear gain in success on the occlusion-prone subset over at least 100 episodes per task, with no loss on the remaining tasks.
The strongest existing monitor is a ContactGuard-style abort. It uses the same multi-view latent predictor with a logistic failure probe trained on the same labeled rollouts. In its own setting it reaches ROC AUC 0.992 on cup grasping and 0.946 on box grasping (ContactGuard). We compare failure-forecast AUC on our data, and we compare task success when its abort is followed by the fixed retry rule. Our method must at least match its detection AUC and exceed its end-to-end success.
The ablations isolate each ingredient. Replacing our learned distance with the gripper-to-target Euclidean distance during waypoint selection tests the core claim. Replacing it with a geometric visibility score tests whether learning adds anything beyond line of sight. That score is the fraction of target pixels visible from each camera at the waypoint, computed from simulator ground truth, as in MobileH2R's visibility objective. Replacing the quasimetric with a symmetric latent distance trained identically, in the style of METRICMM, tests asymmetry. Replacing it with an unconstrained goal-conditioned value regressor of the same size, which has no triangle inequality, tests composition. All variants share the encoder, data, waypoint pool and tuning budget.
Why not just…?
Why not just add more static cameras? AV-ALOHA's static multi-camera setups reached 30% on Thread Needle and 40% on real Hidden Pick, against 52% and 60% with an active camera (Active Vision Might Be All You Need). METRICMM found that naive concatenation of camera streams collapsed under sensor failure (return 0.06 vs 4.55 on Fetch-Slide). RLBench already provides several fixed views, so we will also run all baselines with every available camera. Any remaining gain from recovery is then attributable to choosing better poses, not to adding views.
Why not just penalize occlusion explicitly with a visibility term? This is the real threat. A MobileH2R-style geometric visibility score could select waypoints as well as a learned distance. Our argument is that visibility is necessary but not sufficient. Some visible poses still fail, for example because a fingertip is about to hit a rim, and some partially occluded poses succeed. A distance grounded in success should rank these correctly where pixel visibility cannot. The visibility-score ablation settles this directly. If it matches our method, we will report that the learned metric's value lies only in not needing ground-truth geometry at test time, and the claim narrows accordingly.
Why not just learn a goal-conditioned success or value function? A value function is already a directed notion of distance to success, and for single-step "go here, then resume" decisions it may suffice. The quasimetric structure should matter when waypoints are stitched from different trajectories, because only a distance that obeys the triangle inequality makes the composed cost meaningful across those trajectories. The unconstrained-regressor ablation tests this. If that ablation ties our method, we will drop the structural claim and keep the observation-dependent-distance claim.
Why not just use force sensing for the blind phase? FACT (Demystifying When and Why VLAs Fail) lifts contact-rich success from 41% to 66% with modulated force input. That addresses the in-contact phase. Our target is the pre-contact choice of approach, so the two are complementary rather than competing, and we treat force as out of scope.
Proposed Experiments
The first decisive check tests whether the learned distance captures occlusion at all. We build matched pairs: gripper poses at equal Euclidean distance from the target, sampled at different approach angles, so that ground-truth visibility differs. We then measure whether the learned distance predicts empirical success-from-state, estimated by repeated rollouts from each pose, better than Euclidean distance does. The metric is rank correlation, together with partial correlation after controlling for visibility fraction. A positive partial correlation means the distance captures more than line of sight.
The second check is end-to-end. We report success rate, recovery rate among forecast failures, and the fraction of final-approach frames with the target occluded. These are measured on the occlusion-prone RLBench subset for the base policy, ContactGuard-style abort plus retry, and our recovery layer with each distance ablation, under an equal number of allowed retries.
The third check probes direction and transfer. We report the asymmetry ablation on tasks with irreversible contact effects, such as nudged or toppled objects. As a coarse sanity check of the learned metric in an obstacle setting with existing numbers, we use the Fetch suite with image observations, where G-HGG reaches 80% on FetchPushLabyrinth while HER and Euclidean HGG reach 0% (G-HGG). If budget allows, a small real-robot study on a Franka with wrist and static cameras will repeat the second check on two grasping tasks.
The proposal fails if the geometric visibility score matches our distance in both the first and second checks. It also fails if the recovery gain disappears once all cameras are given to the baselines.
References
- Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation
- Learning Actionable Representations with Goal Conditioned Policies
- Geometry of Uncertainty: Learning Metric Spaces for Multimodal State Estimation in RL
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- MobileH2R: Learning Generalizable Human to Mobile Robot Handover Exclusively from Scalable and Diverse Synthetic Data
- RLBench: The Robot Learning Benchmark & Learning Environment.
- RVT: Robotic View Transformer for 3D Object Manipulation
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- Demystifying When and Why VLAs Fail in Contact-Rich Tasks and How to Fix Them