Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review

Guoguang Du, Kai Wang, Shiguo LianarXiv 2019 · arXiv

Robotic grasping is a fundamental challenge in robotics, bridging the gap between computer vision and physical manipulation. This review provides a comprehensive roadmap of the "pixels-to-pick" pipeline, categorizing the process into four essential tasks: object localization, 6D pose estimation, grasp detection, and motion planning.

The Perception-to-Action Pipeline

To perform a task as simple as picking up a coffee mug, a robot must navigate a complex sequence of operations. In unstructured environments—such as cluttered warehouses or domestic settings—this involves identifying objects, determining their spatial orientation, calculating stable contact points, and planning a collision-free trajectory.

1. Object Localization

The first step is identifying "what" and "where." The field has transitioned from hand-crafted descriptors (SIFT, SURF) to Deep Convolutional Neural Networks (DCNNs).

  • Two-stage methods (e.g., Mask R-CNN) prioritize precision by first proposing regions and then classifying them.
  • One-stage methods (e.g., YOLO, SSD) prioritize speed, enabling real-time performance by predicting bounding boxes and class probabilities in a single pass.

2. 6D Pose Estimation

Understanding an object's 3D translation and rotation (6 degrees of freedom) is vital for precise manipulation. Methods include:

  • Correspondence-based: Matching 2D image pixels to 3D model coordinates.
  • Template-matching: Comparing the observed image against a library of pre-rendered views.
  • Voting-based: Using techniques like Hough forests to aggregate local features into a global pose estimate.

3. Grasp Detection

Once the object is located, the robot must decide where to place its grippers.

  • Analytical Approaches: Rely on "force closure" and "shape closure" physics models to ensure stability.
  • Empirical (Data-Driven) Approaches: Learn from experience. This includes retrieving grasps for known objects, generalizing for familiar shapes, or using deep learning to predict "graspability" from raw depth images for entirely unknown objects.

4. Motion Planning

The final stage converts a grasp point into a physical trajectory. Techniques range from Dynamic Movement Primitives (DMPs) for smooth motion to Imitation Learning and Reinforcement Learning (RL), where robots learn through trial and error or by observing human demonstrations.

Data: The Engine of Modern Grasping

The shift toward deep learning has necessitated massive datasets to train robust models:

  • Localization: PASCAL VOC and MS COCO remain foundational benchmarks.
  • 6D Pose: LineMod is the standard for texture-less objects, while YCB-Video provides a benchmark for cluttered scenes.
  • Grasp Detection: Dex-Net 2.0 is a standout, providing 6.7 million synthetic point clouds to train robust models.
  • Planning: Large-scale real-world datasets, such as Google’s "Grasping" dataset (800,000 attempts), help bridge the gap between theory and practice.

Key Insights and the Future of Manipulation

Modular vs. End-to-End

A modular pipeline (Task 1 → 2 → 3 → 4) offers interpretability but is prone to "error propagation," where a small mistake in localization leads to a total grasp failure. Conversely, end-to-end methods (Pixels → Grasp) are more robust to noise but require significantly more training data.

The Sim-to-Real Gap

Collecting real-world data is prohibitively slow and expensive. Most state-of-the-art models now train on synthetic data (like Dex-Net) and utilize Domain Adaptation to ensure the learned skills transfer to the physical world.

Beyond Geometry: Semantic Perception

Future systems must look beyond simple shape. Understanding material properties (fragility) and weight distribution (center of mass) is essential for handling objects like glass bottles or heavy tools. The paper suggests that estimating these semantic properties is critical for the next generation of robots.

Closing the Loop

Most current systems are "one-shot," meaning they plan a path and execute it blindly. The authors argue for closed-loop control, where continuous visual and tactile feedback allows the robot to adjust its approach in real-time, much like a human does.

Conclusion

While deep learning has propelled robotic grasping forward—evidenced by DenseFusion’s 96.8% accuracy on YCB-Video and Dex-Net’s 93% success rate—challenges remain. Handling extreme occlusion, generalizing to radically new shapes, and operating in dynamic scenes alongside humans represent the next frontier for vision-based robotic manipulation.

Summary updated

Abstract

This paper presents a comprehensive survey on vision-based robotic grasping. We conclude three key tasks during vision-based robotic grasping, which are object localization, object pose estimation and grasp estimation. In detail, the object localization task contains object localization without classification, object detection and object instance segmentation. This task provides the regions of the target object in the input data. The object pose estimation task mainly refers to estimating the 6D object pose and includes correspondence-based methods, template-based methods and voting-based methods, which affords the generation of grasp poses for known objects. The grasp estimation task includes 2D planar grasp methods and 6DoF grasp methods, where the former is constrained to grasp from one direction. These three tasks could accomplish the robotic grasping with different combinations. Lots of object pose estimation methods need not object localization, and they conduct object localization and object pose estimation jointly. Lots of grasp estimation methods need not object localization and object pose estimation, and they conduct grasp estimation in an end-to-end manner. Both traditional methods and latest deep learning-based methods based on the RGB-D image inputs are reviewed elaborately in this survey. Related datasets and comparisons between state-of-the-art methods are summarized as well. In addition, challenges about vision-based robotic grasping and future directions in addressing these challenges are also pointed out.
BibTeX
@misc{du2019vision,
  author = {Guoguang Du and Kai Wang and Shiguo Lian},
  title = {{Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review}},
  year = {2019},
  eprint = {1905.06658},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.06658},
}
Figure