Overcoming Visual Blind Spots in Robotic Manipulation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 21,140 papers1,322 conceptsfrozen 2026-10-04

In robotic manipulation, the robot’s own body is often its greatest perceptual obstacle. To perform a task, a robot must move its end-effector toward a target; yet, the closer it gets, the more likely its own arm or gripper is to block the camera's view. This phenomenon, known as self-occlusion, creates a critical "blind spot" at the very moment high-precision feedback is most needed.

Recent research has shifted from treating occlusion as an unavoidable noise source to treating it as a fundamental geometric constraint that must be managed through active hardware, multimodal sensing, and stateful representation.

The Geometry of the Blind Spot

Self-occlusion manifests differently depending on where the "eyes" are placed. In fixed-camera setups, the robot's arm often sweeps between the lens and the workspace, completely severing the visual feedback loop. Andrew Lee and colleagues note that these static perspectives cannot resolve visual ambiguities when an object is tucked under a shelf or behind a barrier Lee et al..

Conversely, eye-in-hand configurations—where the camera is mounted on the wrist—solve the arm-blockage problem but introduce a new one: the "final approach" occlusion. As the gripper closes in on a peg or a handle, the object itself eventually fills the field of view or is obscured by the gripper's fingers. Ruochen Yin and their team demonstrate that in high-precision tasks like peg-in-hole assembly, this creates a terminal blind spot that forces the robot to execute the final, most delicate phase of the task "blind" Yin et al..

Active Vision and the Actuated Neck

If a static viewpoint is the problem, the most direct solution is to give the robot the ability to "peek" around its own limbs. This is the core intuition behind Active Vision.

Instead of relying on fixed mounting points, researchers like Bipasha Sen and Pulkit Agrawal have introduced human-like actuated necks—multi-degree-of-freedom arms dedicated solely to positioning a camera. By training policies to dynamically adjust their viewpoint, these systems can "look around" occlusions to maintain a clear line of sight Sen et al.. Similarly, the AV-ALOHA system adds a third "vision arm" to a bimanual setup, allowing the robot to actively manage its perspective during complex tasks Lee et al..

Multimodal Fusion as a Perceptual Bridge

When vision is physically blocked, the robot must rely on other senses to "fill in" the missing data. Visual-tactile fusion treats touch as the natural successor to vision during the final centimeters of a grasp.

Fan Zhu and colleagues developed a framework that uses proprioceptive feedback—internal pressure and joint sensing—to maintain spatial awareness when the gripper blocks the camera Zhu et al.. This cross-modal approach is further refined by models that can "imagine" tactile sensations from visual data, or vice versa. Jun-Yan Zhu and Yunzhu Li demonstrated that by learning the correlation between sight and touch, robots can synthesize accurate tactile signals even for regions hidden by self-occlusion Zhu et al..

Incorporating the Self in the Sensor Model

A subtle but critical failure point in robotic navigation is the "self-scanning" gap. If a robot is trained in a simulation where its own body is invisible to its sensors, it will fail in the real world when its LiDAR or depth camera perceives its own limbs as unexpected obstacles.

The Gallant framework, developed by Qingwei Ben and Jingbo Wang, addresses this by explicitly including the robot's physical structure in the sensor's field of view during training. By learning to recognize and ignore its own "self-scans," the robot avoids the domain gap that typically causes navigation failures in complex 3D terrains Ben et al..

Stateful Policies and Temporal Reasoning

Finally, if a robot cannot see an object now, it should remember where it was then. Traditional "reactive" policies, which map the current image directly to an action, are highly vulnerable to temporary occlusions.

To solve this, researchers are moving toward stateful behaviors. Fabian Weigend and Heni Ben Amor introduced Diff-Control, which uses a transition model to make diffusion-based policies aware of previous states. This internal memory allows the robot to maintain task execution even when the environment is momentarily obscured, effectively "bridging" the gap in observation Weigend et al.. Similarly, recurrent architectures like DRQN allow robots to navigate "tactile void spaces" by integrating sensor data over time Yuan et al..

Papers in this version

Showing 60 of 1,140 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 934 papers
  2. Version 1 · 2026-10-04 · 1140 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_visual_occlusion_in_manipulation_61b9fc8d,
  title = {visual occlusion in manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-occlusion-in-manipulation/rdr_61b9fc8dcd984d73afcf94adcc830870},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_overcoming_visual_blind_spots_in_robotic_61b9fc8d_v1,
  title = {Overcoming Visual Blind Spots in Robotic Manipulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-visual-blind-spots-in-robotic-manipulation/rdr_61b9fc8dcd984d73afcf94adcc830870/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}