Generative Models for Multi Modal Robotic Grasping

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 166 papers84 conceptsfrozen 2026-10-04

In traditional robotic manipulation, the goal was often framed as a regression problem: given an image of an object, predict the single "best" coordinates for a gripper to pick it up. However, physical interaction is inherently multi-modal. There is rarely one "best" way to pick up a mug; you can grab the handle, the rim, or the body.

When a standard regression model tries to learn from these multiple valid solutions, it often fails by "averaging" them. If a model sees two valid grasps on opposite sides of a bowl, the average of those two poses might be a gripper positioned directly in the center of the bowl—a physically impossible configuration.

To solve this, researchers have shifted toward generative modeling. Instead of predicting a single point, these models learn the entire probability distribution of valid interactions.

The Shift from Points to Distributions

The core insight of this research direction is that a robot should "imagine" a variety of valid outcomes and then sample from them. Early work in this area, such as 6-DOF Grasping for Target-driven Object Manipulation in Clutter by Arsalan Mousavian and colleagues, utilized Variational Autoencoders (VAEs). By mapping partial point clouds to a latent space of viable 6-DoF configurations, the system could sample diverse, high-quality grasp candidates even when the object was partially obscured.

While VAEs provided a foundation, the field has increasingly moved toward Diffusion Models to handle the high-dimensional complexity of multi-fingered, dexterous hands. As Takuma Yoneda argues in Generative Modeling Perspective for Control and Reasoning in Robotics, treating physical stability as a generative distribution allows robots to propose multiple valid configurations for a single goal, making them far more robust to real-world uncertainty.

Diffusion for Dexterous Complexity

Grasping with a human-like hand is significantly harder than using a simple two-fingered gripper because the "search space" of possible finger joints is massive. Diffusion models excel here because they don't try to find a solution in one step. Instead, they start with random noise and iteratively "denoise" it into a valid hand configuration.

In Bimanual Grasp Synthesis for Dexterous Robot Hands, Yanming Shao and Chenxi Xiao demonstrated that Denoising Diffusion Probabilistic Models (DDPMs) could synthesize complex two-handed grasps for heavy objects by learning the distribution of successful poses from massive datasets. Similarly, DexGrasp-Diffusion by Zhengshen Zhang and others uses this iterative process to generate stable grasps that generalize across different hand shapes and sizes.

Factoring the Problem via Contact Maps

One way to make generative models even more effective is to break the problem into two steps:

  1. Where to touch: Generate a "contact map" on the object's surface.
  2. How to reach: Generate the hand pose that satisfies those contacts.

Haoming Li and colleagues pioneered this in Contact2Grasp. By using a VAE to learn a latent space of potential contact patterns, the robot can first decide how it wants to interact with the object (e.g., a pinch vs. a power wrap) before calculating the specific joint angles. This factorization ensures that the generated grasps are not just mathematically stable, but physically plausible and diverse.

Guiding the Imagination with Language

The most recent frontier in this research is conditioning these generative models on natural language or specific tasks. If a user says "pick up the hammer to hand it to me," the robot shouldn't just find any stable grasp; it needs to avoid the handle so the human can grab it.

Researchers like Minh Nhat Vu and Ngan Le have addressed this by introducing "Negative Prompt Guidance" in diffusion frameworks. In LGrasp6D, they show how a robot can use language to steer its generative process away from distracting objects and toward the specific target requested.

This task-awareness is further refined in work like Task-Oriented Human Grasp Synthesis by An-Lun Liu and Yu-Wei Chao, which uses diffusers to generate poses specifically tailored for downstream actions like stacking or shelving. By modeling the distribution of functional grasps, robots move closer to the versatility of human manipulation.

Papers in this version

Showing 60 of 66 papers.

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generative_models_for_multi_modal_roboti_3c7745ab,
  title = {Generative Models for Multi Modal Robotic Grasping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-models-for-multi-modal-robotic-grasping/rdr_3c7745ab98874f7ab8f20bc5207dc233},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_generative_models_for_multi_modal_roboti_3c7745ab_v1,
  title = {Generative Models for Multi Modal Robotic Grasping},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-models-for-multi-modal-robotic-grasping/rdr_3c7745ab98874f7ab8f20bc5207dc233/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}