Robotic Grasping Under Heavy Occlusion

48 concepts47 papers1 proposal

In a controlled laboratory setting, a robot can easily pick up a lone screwdriver from a flat table. But in a warehouse bin or a kitchen drawer, that same screwdriver might be buried under a pile of pliers, partially obscured by a hammer, or visible only as a small yellow handle. This transition from isolated objects to dense clutter introduces a fundamental challenge: the loss of visual and geometric information due to occlusion.

To operate in the real world, robotic systems must move beyond simple pattern matching and learn to reason about what they cannot see.

The Information Gap in Cluttered Scenes

Standard computer vision models often assume that an object's features—its edges, textures, and overall geometry—are fully visible. In clutter, this assumption fails. As [ten Pas] et al. observe in High precision grasp pose detection in dense clutter, haphazard piles create sensor noise and visibility constraints that make it nearly impossible to differentiate a stable grasp point from an unstable one.

When an object is occluded, the robot faces two distinct problems:

  1. Geometric Uncertainty: The robot cannot see the object's center of mass or the "far side" where a gripper might need to make contact.
  2. Physical Entanglement: Even if a grasp is found, the target might be physically blocked by other items, meaning the robot must reason about the sequence of moves required to "uncover" the target.

Learning from Synthetic Clutter

Because labeling millions of real-world occluded images is impractical, researchers have turned to massive synthetic datasets to bridge the "data bottleneck." By simulating physics and light transport, these systems can generate "ground truth" for what lies beneath the surface.

[Liang] et al. demonstrated this in Dex-Net 2.0 by training models on millions of synthetic point clouds. This approach allows the robot to learn a "Grasp Quality" metric that accounts for the partial information provided by a depth sensor. Similarly, the Jacquard dataset by [Depierre] et al. and MetaGraspNet by [Gilles] et al. provide rich annotations for how objects are stacked, helping models learn to "see through" the clutter by predicting the full shape of partially hidden items.

From Detection to Object-Centric Reasoning

In a pile of overlapping objects, a global view of the scene is often too noisy for precise manipulation. To solve this, researchers have moved toward object-centric or part-aware models.

Instead of analyzing the whole bin at once, the ROI-GD framework focuses on specific Regions of Interest (ROIs). By isolating individual objects from the background clutter, the system can associate specific grasp points with specific items, even when their geometries overlap. This is mirrored in the work of [Xu] et al. in Part-Aware Refinement Network for Occlusion Vehicle Detection, which uses the confidence of visible "parts" to infer the presence of a partially hidden whole.

Active Perception and Interaction

Sometimes, no amount of reasoning can resolve the ambiguity of a dense pile. In these cases, the robot must transition from a passive observer to an active participant.

Systems like INVIGORATE, proposed by [Zhang] et al., treat perception as a "noisy observation" within a Partially Observable Markov Decision Process (POMDP). If the robot is unsure which object is the target, it doesn't just guess; it can ask a human for clarification or physically move an obstacle to get a better view.

This concept of active view planning is further refined in VISO-Grasp by [Wen] et al., where the robot uses a Vision-Language Model (VLM) to reason about where a target might be hidden and then moves its camera to "uncover" the object. This shifts the paradigm from "grasping what is seen" to "interacting to see what must be grasped."

Grounding Language in the Mess

The challenge of occlusion is even steeper when a robot must follow natural language instructions, such as "pick up the bottle behind the box." In OCID-Ref, [Wang] et al. highlight that standard visual grounding fails in clutter because objects are often touching or stacked.

To resolve this, [Shridhar] and [Hsu] developed INGRESS, which uses a "grounding-by-generation" approach. By generating its own descriptions of the scene and comparing them to the user's input, the robot can identify which specific, partially-obscured object the user is referring to, effectively mapping linguistic spatial relationships (like "under" or "behind") to the physical reality of the clutter.

Key Foundational Work

TINA: The Sheffeild AIVRU Vision System (1987) — contributed 1 concept elements to this direction.
Seeing Behind Occlusions (1994) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_clutter_induced_occlusion_8af4737d,
  title = {clutter-induced occlusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/clutter-induced-occlusion/rdr_8af4737dc415436ab609c78ac04a08b4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_clutter_induced_occlusion_8af4737d_v1,
  title = {clutter-induced occlusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/clutter-induced-occlusion/rdr_8af4737dc415436ab609c78ac04a08b4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake