In traditional robotic manipulation, the goal was often framed as a regression problem: given an image of an object, predict the single "best" coordinates for a gripper to pick it up. However, physical interaction is inherently multi-modal. There is rarely one "best" way to pick up a mug; you can grab the handle, the rim, or the body.
When a standard regression model tries to learn from these multiple valid solutions, it often fails by "averaging" them. If a model sees two valid grasps on opposite sides of a bowl, the average of those two poses might be a gripper positioned directly in the center of the bowl—a physically impossible configuration.
To solve this, researchers have shifted toward generative modeling. Instead of predicting a single point, these models learn the entire probability distribution of valid interactions.
The Shift from Points to Distributions
The core insight of this research direction is that a robot should "imagine" a variety of valid outcomes and then sample from them. Early work in this area, such as 6-DOF Grasping for Target-driven Object Manipulation in Clutter by Arsalan Mousavian and colleagues, utilized Variational Autoencoders (VAEs). By mapping partial point clouds to a latent space of viable 6-DoF configurations, the system could sample diverse, high-quality grasp candidates even when the object was partially obscured.
While VAEs provided a foundation, the field has increasingly moved toward Diffusion Models to handle the high-dimensional complexity of multi-fingered, dexterous hands. As Takuma Yoneda argues in Generative Modeling Perspective for Control and Reasoning in Robotics, treating physical stability as a generative distribution allows robots to propose multiple valid configurations for a single goal, making them far more robust to real-world uncertainty.
Diffusion for Dexterous Complexity
Grasping with a human-like hand is significantly harder than using a simple two-fingered gripper because the "search space" of possible finger joints is massive. Diffusion models excel here because they don't try to find a solution in one step. Instead, they start with random noise and iteratively "denoise" it into a valid hand configuration.
In Bimanual Grasp Synthesis for Dexterous Robot Hands, Yanming Shao and Chenxi Xiao demonstrated that Denoising Diffusion Probabilistic Models (DDPMs) could synthesize complex two-handed grasps for heavy objects by learning the distribution of successful poses from massive datasets. Similarly, DexGrasp-Diffusion by Zhengshen Zhang and others uses this iterative process to generate stable grasps that generalize across different hand shapes and sizes.
Factoring the Problem via Contact Maps
One way to make generative models even more effective is to break the problem into two steps:
- Where to touch: Generate a "contact map" on the object's surface.
- How to reach: Generate the hand pose that satisfies those contacts.
Haoming Li and colleagues pioneered this in Contact2Grasp. By using a VAE to learn a latent space of potential contact patterns, the robot can first decide how it wants to interact with the object (e.g., a pinch vs. a power wrap) before calculating the specific joint angles. This factorization ensures that the generated grasps are not just mathematically stable, but physically plausible and diverse.
Guiding the Imagination with Language
The most recent frontier in this research is conditioning these generative models on natural language or specific tasks. If a user says "pick up the hammer to hand it to me," the robot shouldn't just find any stable grasp; it needs to avoid the handle so the human can grab it.
Researchers like Minh Nhat Vu and Ngan Le have addressed this by introducing "Negative Prompt Guidance" in diffusion frameworks. In LGrasp6D, they show how a robot can use language to steer its generative process away from distracting objects and toward the specific target requested.
This task-awareness is further refined in work like Task-Oriented Human Grasp Synthesis by An-Lun Liu and Yu-Wei Chao, which uses diffusers to generate poses specifically tailored for downstream actions like stacking or shelving. By modeling the distribution of functional grasps, robots move closer to the versatility of human manipulation.