Self-Occlusion Is Ignorable Given Proprioception: Mask-Aware Visuomotor Policies

When a robot reaches for an object, its own arm and gripper often block the camera's view of the target, and this happens in the last few centimeters, where precision matters most. Current fixes add hardware, such as extra cameras, a second arm that moves the camera, or touch sensors. Alternatively, they hope a policy learns to cope with pixels that show the robot's arm where the target used to be. We start from a simple observation from statistics on missing data. The robot always knows which parts of the scene it cannot see, because the silhouette of its own body in each camera follows exactly from its joint angles, its geometric model, and the camera calibration. Missing-data theory separates data that is missing at random given what you observe from data that is missing because of its own hidden value. Under the first condition, a model can handle the gaps correctly without guessing why they are missing, but only if it is told where the gaps are. Self-occlusion falls into the first category once joint angles are taken into account. Today's visuomotor policies are never told where the gaps are. They see arm pixels and cannot tell "the object is not here" from "I cannot see here."

We propose to render the robot's self-occlusion mask from its kinematics at every control step. Image or point tokens behind that mask become explicitly missing. Each missing token is replaced by a stored memory of the last time that region was visible, plus a tag saying how long ago that was. The policy is trained with occlusions transplanted from real arm configurations rather than random patches. The test case is concrete: can this software change recover, from a static-camera setup, the gain that AV-ALOHA gets from a dedicated 7-joint camera arm? On the AV-ALOHA Thread Needle task, that gain is 52% versus 30%.

Core Claim

Let $x$ be the full scene content seen by a camera, split into visible content $x_{\text{obs}}$ and hidden content $x_{\text{mis}}$. Let $q$ be the robot's joint configuration, and let $m$ be the binary mask of image regions hidden behind the robot's body. Because $m = S(q)$, where $S$ renders the robot's silhouette from its geometric model and the camera calibration, we have

$$ P(m \mid x_{\text{obs}}, x_{\text{mis}}, q) = P(m \mid q), $$

so self-occlusion is missing at random given proprioception. The research file's own gap analysis frames manipulator occlusion as missing-not-at-random. That label is accurate only for a model that ignores $q$, and it would point toward heavy generative imputation of the kind used by RefiDiff, a diffusion-based tabular imputer evaluated under that harder regime. We argue the cheaper, principled route is to make the mask explicit.

This claim has a testable consequence. Mask-aware handling should help a lot on self-occlusion. It should help much less on occlusion of the target by other objects, which really is missing-not-at-random because the mask depends on unobserved object poses. If we observe that asymmetry, the mechanism is the one we describe, not generic regularization.

The formulation also offers a way to resolve a tension in the literature. ContactGuard runs a predictive latent world model to forecast grasp failure before contact. It found that feeding joint states into its model became a shortcut that degraded visual representations when data was scarce. DECO, a diffusion transformer with a plug-in tactile adapter, found joint-state conditioning essential. Our design uses $q$ through a fixed geometric renderer rather than as a learned input. The policy gets the information proprioception carries about visibility without a learned shortcut path.

Research Program

The system has three parts.

The first is a mask renderer. At each control step it rasterizes the robot's geometric model at configuration $q$ into every camera, using the same calibration that 3D methods like RVT already require. The result is a per-pixel occlusion mask and, with depth, a per-ray "unknown beyond here" marker.

The second is a mask-aware token layer that sits in front of an existing policy backbone. Any image patch more than a threshold fraction hidden is marked as missing. Its feature is replaced by the last feature observed for that region, and the token carries a learned embedding of the missing indicator and of the time elapsed since the region was last seen. For static cameras, a "region" is a fixed patch location. For wrist cameras, last-seen features are stored as back-projected 3D points and re-projected using the known camera motion. For point-cloud policies, the analogue is three-state occupancy (free, occupied, unknown), where unknown is assigned by casting rays past the rendered arm.

This layer borrows its stance from imputation-free missing-data models. AnyNets encodes only observed features and pools them as a set, keeping 0.913 clustering accuracy on Breast Cancer at 70% missingness where zero-filling fell to 0.621. Flex-MoE learns embeddings indexed by which modalities are present. Both were built for static clinical records. We transfer the idea of an explicit, learnable missingness signal to streaming image tokens, where missingness changes every frame and stale values carry real information.

The third part is occlusion-transplant training. Demonstration frames are augmented by rendering arm silhouettes from joint configurations sampled from other timesteps and trajectories, then compositing them into the image and the mask. The policy therefore sees the spatially structured, task-correlated occlusions it will meet at deployment. This differs on purpose from the uniform spatial masking in DynaRend, which pretrains a 3D triplane model by reconstructing randomly masked views. Uniform masking teaches reconstruction of arbitrary holes. It does not teach that holes appear exactly where the gripper is about to act. We isolate the masking distribution as its own variable.

We avoid a generative imputer on purpose. RISE, which pairs neural processes with posterior estimation, notes that Gaussian latent assumptions struggle with multimodal missing content. Occluded object poses during grasping are exactly that kind of content. If the ignorability argument holds, the policy needs a correct missingness signal and memory, not a sampled guess at the hidden pixels.

Proposed Experiments

The primary testbed is RLBench, where segmentation and robot geometry are available in simulation, so exact masks cost nothing. We use the 18-task subset on which RVT, a model that re-renders point clouds into virtual views, reports 62.9% average success and trains in about 10 hours. That makes repeated ablations affordable. We run two camera regimes: the standard multi-camera setup, and a degraded front-plus-wrist setup in which terminal self-occlusion is frequent. We add a stratified analysis that bins every evaluation step by the fraction of the target hidden behind the arm.

The second testbed is the AV-ALOHA task suite in its static-camera configuration (Thread Needle, Occluded Insertion, Pour Test Tube, Hidden Pick). There we train Action Chunking with Transformers with and without the mask-aware layer, using the same demonstrations.

A real-world check on a Franka with a wrist camera follows the reactive-grasping protocol of VFAS-Grasp, including the moving turntable, where open-loop terminal execution reached only 35%. Masks there come from the robot's geometric model and calibration, and we deliberately perturb the calibration to measure sensitivity.

Metrics are task success rate, success restricted to episodes with heavy terminal occlusion, and a diagnostic probe. The probe is a linear readout of target pose from the policy's latent at steps where the target is fully hidden. It tests whether the latent tracks the hidden object or forgets it.

Baselines

The naive reference is the unmodified backbone (RVT on RLBench, ACT on AV-ALOHA) seeing raw frames with arm pixels left in place, as in the original papers. Every other variant shares its data, training steps, and hyperparameter search budget. We tune each variant with the same number of trials over learning rate and augmentation strength.

The strongest temporal competitor is the same backbone given a stacked history of past frames equal in length to our memory horizon. It has the same raw information about where the object was before occlusion, so it is the fairest test of whether explicit masks add anything. Beating it means a statistically significant gain on heavily occluded episodes, not merely on the average.

The strongest masking competitor is uniform random patch masking during training at the same masking ratio as our transplant augmentation, standing in for the DynaRend-style objective. Beating it shows that a kinematically matched masking distribution matters.

Two ablations isolate the key ingredient. The first gives the mask as an extra input channel without memory replacement. The second uses last-seen memory without the missing indicator and age tag. If either matches the full method, the corresponding component is unnecessary.

For context, we compare against reported numbers from methods that change hardware or architecture. On RLBench, DynaRend reports 76.6% average success over 71 tasks, and RVT reports 62.9% over 18. On AV-ALOHA, the active camera arm reports 52% on Thread Needle versus 30% static, and 60% versus 40% on real Hidden Pick. Our headline target is closing at least half of the static-to-active gap on the AV-ALOHA occlusion tasks with no added hardware. For the tactile alternative, DECO reports 82.50% with its tactile adapter against 72.25% vision-only on DECO-50. We cannot rerun that humanoid setup, so it stays a reference point, not a head-to-head comparison.

Why not just…?

Why not just give the policy a longer history of frames? This is the most serious threat. The information about the pre-occlusion target pose is technically present in past frames. Our bet is that a policy seeing arm pixels cannot tell which past frames are still valid for which regions, and that this gets worse as history grows and the arm sweeps through the view. The matched-history baseline settles this directly. If frame stacking equals the full method on heavily occluded episodes, we will report that the contribution reduces to memory and drop the missingness framing.

Why not just add cameras or an active camera arm? AV-ALOHA shows that an active camera helps on occlusion-heavy tasks. It also shows that the 21-joint action space hurts on simple tasks, and the extra arm is costly and physically bulky. VFAS-Grasp further shows that even wrist cameras go blind as the target enters the jaws. More cameras shrink the blind region but cannot remove it. Our proposal is complementary, and the AV-ALOHA comparison measures how much of the hardware gain software alone recovers.

Why not just use touch once vision fails? Tactile sensing pays off after contact. Touch in the Wild raised occluded test-tube collection from 25% to 85% with a visuo-tactile gripper. But ContactGuard documents that many failures happen in the pre-contact approach, where touch is silent. Our stratified analysis separates pre-contact from post-contact steps, and we expect our gains to concentrate in the pre-contact window. If they do not, touch is the better investment for those tasks.

Why not just train with random masking? Uniform masking is cheap and already used by DynaRend. It is a real competitor if occlusion structure does not matter. The matched-ratio uniform-masking baseline tests this. Our prediction is that transplanted arm silhouettes help specifically in the terminal phase, because that is where real occlusions concentrate.

Decisive Checks

Three results would confirm or kill the idea.

First, gains should grow with the fraction of the target hidden by the arm and vanish when nothing is hidden. A flat improvement across bins would indicate generic regularization, not missing-data handling.

Second, the predicted asymmetry should appear. On RLBench variants where the target is hidden by a distractor object rather than the arm, the mask-aware layer has no correct mask to use, so its advantage over frame stacking should shrink sharply. If it helps equally there, our ignorability explanation is wrong.

Third, the pose probe should decode the hidden target more accurately than all baselines during full occlusion. Success should also degrade gracefully when masks are dilated or the calibration is perturbed by a few millimeters, which addresses the calibration sensitivity RVT already acknowledges.