A robot that manipulates by looking loses its view at exactly the wrong moment. As the hand closes on an object, the robot's own fingers and arm block the cameras, and the policy has to act with little useful visual information just before contact. The standard remedy is to add a second kind of sensing, such as touch pads, force or joint torque, and fuse it with vision. Current systems fuse these streams in a way that does not depend on how much of the scene the cameras can currently see. Yet the robot already knows the answer. It has a geometric model of its own body, calibrated cameras, and joint encoders, so it can render at every timestep which pixels near its fingertips are hidden behind its own links. That rendering is essentially free and needs no labels.
This note proposes using that rendered self-occlusion map as the explicit control signal for fusion. Each camera's features are weighted by how visible the fingertip region is in that camera. Touch and force features are weighted by how blind the cameras are overall. The streams are combined by simple weighted addition rather than cross-attention, and during training the touch branch's gradients are protected when vision is blocked. The claim is specific and testable. An occlusion signal computed from the robot's body model beats fusion weights that a network learns from features or proprioception, and the advantage should grow when cameras are moved to positions not seen in training, because the geometric signal is recomputed from the new calibration while a learned gate is not.
Where this sits relative to prior work
Two recent papers come closest, and each supplies half of the design. BFA predicts one scalar importance weight per camera with a small multilayer perceptron and fuses views by weighted addition. On a bimanual ALOHA setup it raised average success from 32% for standard ACT to 78%. Its weights are supervised by task-phase labels that GPT-4o generates offline from heuristics, and it fuses only camera views. DECO shows that dexterous hands occlude cameras and that naive concatenation lets dense visual tokens swamp sparse touch signals. It adds touch through a separately trained cross-attention adapter, but that adapter is applied the same way whether or not vision is blocked. We take BFA's cheap additive gating and DECO's non-visual contact stream. We replace BFA's phase labels with a geometric visibility computation, and we replace DECO's fixed conditioning with weights that follow visibility.
Two further papers constrain the design. When would Vision-Proprioception Policies Fail shows that low-dimensional signals win the optimization race and starve the visual branch. Its fix, GAP, scales down proprioceptive gradients during motion transitions detected by offline change-point detection plus an LSTM. ContactGuard found that adding joint states to its latent world model hurt failure prediction in low-data regimes because the model used kinematics as a shortcut, and it mean-pools camera views regardless of occlusion. Together these results warn that a fusion gate fed raw proprioception is likely to shortcut. That motivates computing visibility outside the network rather than asking the network to infer it.
The fusion operator comes from speech. Attention Is Not Always the Answer found that element-wise addition of handcrafted and pretrained features beat multi-head cross-attention in most encoder pairings for voice activity detection, with fewer parameters. We treat this as a hypothesis to test in manipulation, not a settled fact.
Active and virtual-view methods attack the same problem by moving the camera instead of fusing. ActiveVLA scores virtual viewpoints from point clouds and zooms in, and RVT re-renders orthographic views. These methods cannot see surfaces that no physical sensor observed. ActiveVLA also doubles inference latency, from 0.26 s to 0.53 s. They are complementary to our approach rather than replaced by it.
Core mechanism
For camera $i$ at time $t$, we use forward kinematics, the robot's link meshes, and the camera calibration to render the robot body into the image. We also project a fixed region ahead of the fingertips, a small cone where contact will happen. The visibility $v_{i,t} \in [0,1]$ is the fraction of that region's pixels that are not covered by the robot's own links. When depth is available we optionally extend this by also counting fingertip-region pixels whose measured depth is well in front of the rendered fingertip, which catches occlusion by the object or clutter.
The fused feature is
$$\hat f_t = \sum_i \alpha_i(v_{i,t})\, f_{i,t} + \beta\!\left(1-\max_i v_{i,t}\right) g_t,$$
where $f_{i,t}$ is camera $i$'s pooled feature, $g_t$ is the projected touch and force feature, and $\alpha_i$ and $\beta$ are small monotone learned functions (a scalar and a bias passed through a sigmoid). Proprioception enters separately through adaptive layer normalization, as in DECO.
Two training choices protect the contact stream. First, the gradient flowing into the touch encoder is multiplied by $1 + \lambda(1-\max_i v_{i,t})$. This is the mirror image of GAP: GAP scales gradients by motion phase, while we scale them by measured blindness. Second, visibility-aware view dropout zeroes a camera's features with probability rising as its $v_{i,t}$ falls. This teaches the policy that occluded views are unreliable without inventing occlusions that never happen.
Research Program
The work has three parts.
The first is a visibility module. It renders robot masks from URDF meshes and calibration at control rate on the GPU. We will validate it against hand-annotated occlusion masks on a few hundred frames, then release it as a policy-agnostic preprocessing step.
The second is the fusion head above. It will be dropped into ACT and Diffusion Policy on arm platforms and into the DECO architecture on the humanoid.
The third is a diagnostic study that separates three things: whether the occlusion signal is explicit, which fusion operator is used, and whether gradient protection is present. The goal is a clear answer to the question the gaps raise: is the problem missing information about when vision fails, or insufficient fusion capacity?
Baselines
The naive reference point is plain concatenation of all camera, proprioceptive and touch features into ACT or Diffusion Policy. On ALOHA-style multi-view tasks, BFA reports 32% average success for standard ACT. A vision-only variant with no contact stream is included as a second floor.
The strongest existing method on the visuo-tactile question is DECO with its tactile adapter. It reports 82.5% average success on its real-world tasks, against 72.25% without touch (DECO). We will retrain DECO and our variant on identical DECO-50 splits, using the same vision-action pretrained backbone, the same LoRA budget of about 8M trainable parameters for the new modules, and equal hyperparameter sweeps. Offline, we compare action prediction error stratified by visibility. On hardware, we compare real rollouts matched in count. Beating DECO means a statistically significant gain on the lowest-visibility third of contact phases, with no loss on high-visibility phases.
For multi-view weighting, the competitor is a BFA reimplementation trained with its GPT-4o phase labels. Its reported ALOHA figure is 78% average success. We also run a version whose score network is supervised by our visibility values instead, which separates the effect of the label source from the effect of weighted addition.
GAP (When would Vision-Proprioception Policies Fail) is run as reported on Meta-World, where GAP reaches 94.2% on assembly versus 74.6% for standard fusion. We then add our visibility-scaled gradient rule to test whether it is complementary to or redundant with phase-scaled gradients.
On RLBench we cite ActiveVLA's reported 91.8% average over 18 tasks, and 65.9% on COLOSSEUM against a previous best of 64.0%. We do not claim to beat these outright. The comparison is success per unit latency, and success on the task subset where fingertip visibility drops below a threshold.
The key ablations are these. We swap the gate input among geometric visibility, raw proprioception, and learned features only. We replace additive fusion with cross-attention of matched parameter count. We remove gradient protection and remove visibility-aware dropout.
Why not just…?
Why not just concatenate everything and train with random modality dropout? Random dropout teaches robustness to missing views on average, but it does not tell the policy which view is missing now. It also leaves the imbalance DECO and GAP document untouched: dense vision still drowns touch, and proprioception still shortcuts. The concatenation-plus-uniform-dropout arm is in the ablations. If it matches our method on low-visibility phases, that would show the explicit signal adds nothing, and we would report it.
Why not just let a learned gate, BFA-style or mixture-of-experts style as in PaM, figure out occlusion from the features or from joint states? This is the most serious threat. With fixed cameras, visibility is a deterministic function of joint angles, so a gate that sees proprioception could in principle learn it. Two things argue against it in practice. The function is highly nonlinear and must be learned from a few hundred demonstrations. And both GAP and ContactGuard show that proprioceptive inputs get exploited as shortcuts. The decisive test is the gate-input ablation, especially under COLOSSEUM's camera-pose perturbations and a real camera relocation. There, our visibility signal is recomputed from the new calibration, while a gate learned from proprioception encodes the old camera geometry. If the learned gate matches ours even after cameras move, the proposal reduces to "use BFA with touch," and we will say so.
Why not just use cross-attention, as DECO does, which can attend away from occluded views on its own? The speech evidence (Attention Is Not Always the Answer) and BFA's result, which beat concatenated ACT with fewer FLOPs (12.96G versus 16.34G), suggest the extra capacity is not what is missing. But manipulation is not voice activity detection, so we test cross-attention at matched parameters, with and without the visibility signal as an extra token. This separates the effect of the operator from the effect of the information it receives.
Why not just add cameras or re-render virtual views? More cameras reduce occlusion by the arm but not the terminal blind spot between the fingers and the object, which wrist cameras hit by construction (Active Vision Might Be All You Need, VFAS-Grasp). Virtual re-projection pays latency and depends on the depth sensor. We include a condition with an added camera on robosuite to measure how much of the low-visibility gap it closes compared with our fusion.
Proposed Experiments
Simulation comes first. On RLBench and robosuite, visibility is exact because meshes and calibration are known. We use eye-in-hand and over-the-shoulder cameras, plus gripper and joint-force channels as the contact stream. Episodes are stratified by the minimum fingertip visibility reached before contact, and success is reported per visibility stratum, not only as an average.
Generalization is tested on COLOSSEUM's camera-pose and object perturbations, trained on unperturbed demonstrations. The prediction is that our method's margin over learned gates widens specifically on the camera-pose axis.
The visuo-tactile core runs on DECO-50. Offline, we measure action error by visibility stratum. We then run real rollouts on the lid-closing and assembly tasks, where DECO reports its largest touch gain of about 20%. Hardware permitting, a matched ALOHA replication is run against BFA.
The decisive checks are three. The first is whether our method beats the proprioception-gated variant on the lowest-visibility stratum after cameras are relocated. The second is whether additive fusion is at least as good as matched cross-attention there. The third is whether fusion weights plotted over time track the rendered visibility and correlate with where DECO's fixed adapter fails. If the first check fails, the geometric signal is not needed, and the contribution shrinks to the benchmark stratification and the fusion-operator study. That outcome is still useful, and it is the reason the ablation is designed to be clean.
References
- BFA: Best-Feature-Aware Fusion for Multi-View Fine-Grained Manipulation
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- When would Vision-Proprioception Policies Fail in Robotic Manipulation?
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- Attention Is Not Always the Answer: Optimizing Voice Activity Detection with Simple Feature Fusion
- ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation
- RVT: Robotic View Transformer for 3D Object Manipulation
- Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio Encoders
- Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation
- VFAS-Grasp: Closed Loop Grasping with Visual Feedback and Adaptive Sampling