Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps

Chaozheng Wu, Jian Chen, Qiaoyu Cao, Jianchi Zhang, Yunxin Tai, Lin Sun, Kui Jia

Abstract: Learning robotic grasps from visual observations is a promising yet challenging task. Recent research shows its great potential by preparing and learning from large-scale synthetic datasets. For the popular, 6 degree-of-freedom (6-DOF) grasp setting of parallel-jaw gripper, most of existing methods take the strategy of heuristically sampling grasp candidates and then evaluating them using learned scoring functions. This strategy is limited in terms of the conflict between sampling efficiency and coverage of optimal grasps. To this end, we propose in this work a novel, end-to-end Grasp Proposal Network (GPNet), to predict a diverse set of 6-DOF grasps for an unseen object observed from a single and unknown camera view. GPNet builds on a key design of grasp proposal module that defines anchors of grasp centers at discrete but regular 3D grid corners, which is flexible to support either more precise or more diverse grasp predictions. To test GPNet, we contribute a synthetic dataset of 6-DOF object grasps; evaluation is conducted using rule-based criteria, simulation test, and real test. Comparative results show the advantage of our methods over existing ones. Notably, GPNet gains better simulation results via the specified coverage, which helps achieve a ready translation in real test.

1. Introduction

Determining a 6-degree-of-freedom (6-DOF) grasp pose from a single, partial camera view is a fundamental challenge in robotics. Traditional approaches often rely on heuristic sampling of hundreds of candidates followed by neural network scoring. This process is computationally expensive and often fails to cover optimal grasps. Generative models, while faster, frequently suffer from "mode collapse," where predicted grasps cluster around a single point (like the center of mass), lacking the diversity required for complex environments.

We introduce the Grasp Proposal Network (GPNet), an end-to-end solution that leverages the concept of "anchors" from 2D object detection to generate diverse and accurate 6-DOF grasps. By defining potential grasp centers on a regular 3D grid, GPNet ensures broad coverage of the object's surface and geometry.

2. Grasp Proposal Network Architecture

The GPNet architecture consists of three primary stages: point-wise feature extraction, a grasp proposal module, and multi-headed output prediction.

The GPNet architecture features a Grasp Proposal Module that utilizes 3D grid anchors and PointNet++ for localized feature extraction.

The network first processes the input point cloud through PointNet++ to obtain point-wise features. The Grasp Proposal Module then defines a 3D grid of anchors. Observed surface points are paired with these anchors to form initial proposals. To manage the proposal space, geometric pruning is applied to remove physically improbable candidates.

A key innovation is the use of anchor-dependent features. Instead of global context, the network extracts features based on the relative geometry between surface points and the specific grid anchor. This localized context significantly improves the accuracy of the subsequent regression and classification heads:

  • Antipodal Classifier: Determines the physical validity of the grasp (ensuring it won't slip).
  • Grasp Regressor: Predicts fine-grained offsets for the 3D center and orientation.
  • Grasp Classifier: Provides a final confidence score for the proposed grasp.

3. Anchor-Driven Diversity

The anchor-based approach inherently promotes grasp diversity. By distributing potential centers across a rigid grid, the network is forced to explore the entire spatial volume of the object. This prevents the model from converging on a single "best" grasp and provides the robot with multiple options, which is critical when certain approach angles are obstructed by kinematics or obstacles.

Visualization of grasp proposal distributions on a mug under different anchor configurations, demonstrating how grid-based anchors improve coverage and diversity.

4. Experimental Results

GPNet was trained on a large-scale synthetic dataset containing 22.6 million annotated grasps across 226 CAD models. The dataset labels are derived from physics-based simulations in PyBullet, accounting for friction, gravity, and collisions.

In simulation, GPNet achieved a 90% success rate for the top 10% of predictions, significantly outperforming the 6-DOF GraspNet baseline (80% with refinement and only 43.3% without). Crucially, GPNet's coverage rate reached 30.8%, nearly double that of the baseline (16.8%). Real-world experiments on a UR5 robot arm confirmed these findings, with GPNet achieving an 85% success rate on novel objects, demonstrating robust generalization from synthetic training data.

5. Conclusion

GPNet offers a highly efficient, end-to-end framework for robotic grasping. By integrating 3D grid anchors, the model achieves a superior balance of accuracy and diversity compared to heuristic sampling or purely generative methods. Future work will focus on extending this approach to cluttered environments and incorporating real-time physical property inference directly from visual or tactile feedback.

Summary updated

Abstract

Learning robotic grasps from visual observations is a promising yet challenging task. Recent research shows its great potential by preparing and learning from large-scale synthetic datasets. For the popular, 6 degree-of-freedom (6-DOF) grasp setting of parallel-jaw gripper, most of existing methods take the strategy of heuristically sampling grasp candidates and then evaluating them using learned scoring functions. This strategy is limited in terms of the conflict between sampling efficiency and coverage of optimal grasps. To this end, we propose in this work a novel, end-to-end \emph{Grasp Proposal Network (GPNet)}, to predict a diverse set of 6-DOF grasps for an unseen object observed from a single and unknown camera view. GPNet builds on a key design of grasp proposal module that defines \emph{anchors of grasp centers} at discrete but regular 3D grid corners, which is flexible to support either more precise or more diverse grasp predictions. To test GPNet, we contribute a synthetic dataset of 6-DOF object grasps; evaluation is conducted using rule-based criteria, simulation test, and real test. Comparative results show the advantage of our methods over existing ones. Notably, GPNet gains better simulation results via the specified coverage, which helps achieve a ready translation in real test. We will make our dataset publicly available.
BibTeX
@inproceedings{wu2020grasp,
  author = {Chaozheng Wu and Jian Chen and Qiaoyu Cao and Jianchi Zhang and Yunxin Tai and Lin Sun and Kui Jia},
  title = {{Grasp Proposal Networks: An End-to-End Solution for Visual Learning of Robotic Grasps}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=H2zcpHa2Oxq},
}
Figure