Constrained Generative Sampling of 6-DoF Grasps

Jens Lundell, Francesco Verdoja, Tran Nguyen Le, Arsalan Mousavian, Dieter Fox, Ville Kyrki

Abstract

Modern robotic grasping systems often struggle with task-specific requirements, such as picking up a tool by its handle rather than its functional end. We present the Variational Constrained Grasp Sampler (VCGS), a deep generative model that enables robots to focus grasping attempts on specific, user-defined areas of an object. By training on the massive new CONG dataset, VCGS achieves a 75% success rate in real-world trials—a 15% improvement over state-of-the-art unconstrained samplers—while being up to three times more sample-efficient.

Introduction

Most modern robotic grasping research focuses on "unconstrained" sampling: finding any stable, collision-free grasp on an object. While effective for simple pick-and-place tasks, this approach fails when specific parts of an object must be targeted for functional reasons, such as grasping a bottle by its body to squeeze it or a screwdriver by its handle to use it.

The end-to-end pipeline of the VCGS system, from initial setup and segmentation to the execution of a constrained grasp.

Existing solutions typically rely on inefficient filtering—generating thousands of random grasps and discarding those outside the target area—or rigid semantic models that only recognize specific categories like "handles." VCGS offers a general-purpose, task-agnostic alternative that can focus on any area designated by a user or a high-level planner.

Methodology

The core of the system is a Conditional Variational Autoencoder (CVAE) built on a PointNet++ backbone. Unlike standard samplers that only process the object's geometry, VCGS accepts the point cloud along with a point-wise binary feature. This feature acts as a spatial "highlight," informing the network which points belong to the target area.

Illustration of the constrained sampling process: (a) object mesh, (b) point cloud with query point, (c) highlighted target area, and (d) generated 6-DoF grasps focused on that area.

A key insight of this research is that providing the entire object point cloud (with the target area highlighted) is superior to providing only the target area. The global context allows the robot to understand the object's overall geometry and avoid collisions with non-target parts. During inference, the model draws samples from a latent space and decodes them into 6-Degree of Freedom (6-DoF) grasp poses conditioned on the highlighted region.

The CONG Dataset

To train such a model, the authors introduced CONG (CONstrained Grasping), a dataset of unprecedented scale for this field. While previous constrained datasets were limited to a few hundred objects and specific tasks, CONG contains over 14 million training samples across 2,889 unique objects. It was constructed by expanding the ACRONYM dataset using a query-point method to identify all valid grasps within randomly sampled radii on object surfaces.

Experimental Results

The researchers benchmarked VCGS against GraspNet, a leading unconstrained sampler, using both Isaac Gym simulations and a physical Franka Emika Panda robot.

  • Success Rate: In real-world tests, VCGS achieved a 75% success rate, significantly higher than the baseline's 59.4%.
  • Efficiency: VCGS proved to be 1.8x to 3x more sample-efficient. It required an average of 360 samples to find a viable grasp, compared to 660 for the baseline.
  • Inference Speed: By generating high-quality, constrained grasps directly, the system avoids the heavy computational cost of generating and then filtering tens of thousands of unconstrained candidates.

Conclusion and Future Work

VCGS demonstrates that incorporating spatial constraints directly into the generative process leads to more reliable and efficient robotic manipulation. Current limitations include a lack of explicit orientation constraints (e.g., "grasp from the side") and the potential for the sampler to propose grasps that are kinematically unreachable for specific robot arms. Future iterations aim to integrate robot kinematic limits and orientation-based constraints directly into the sampling architecture.

Summary updated

Abstract

Most state-of-the-art data-driven grasp sampling methods propose stable and collision-free grasps uniformly on the target object. For bin-picking, executing any of those reachable grasps is sufficient. However, for completing specific tasks, such as squeezing out liquid from a bottle, we want the grasp to be on a specific part of the object's body while avoiding other locations, such as the cap. This work presents a generative grasp sampling network, VCGS, capable of constrained 6-Degrees of Freedom (DoF) grasp sampling. In addition, we also curate a new dataset designed to train and evaluate methods for constrained grasping. The new dataset, called CONG, consists of over 14 million training samples of synthetically rendered point clouds and grasps at random target areas on 2889 objects. VCGS is benchmarked against GraspNet, a state-of-the-art unconstrained grasp sampler, in simulation and on a real robot. The results demonstrate that VCGS achieves a 10-15% higher grasp success rate than the baseline while being 2–3 times as sample efficient. Supplementary material is available on our project website.
BibTeX
@inproceedings{lundell2023constrained,
  author = {Jens Lundell and Francesco Verdoja and Tran Nguyen Le and Arsalan Mousavian and Dieter Fox and Ville Kyrki},
  title = {{Constrained Generative Sampling of 6-DoF Grasps}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=gZwFIO1QP9},
}
Figure