Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review

Guoguang Du, Kai Wang, Shiguo Lian, Kaiyong ZhaoArtif. Intell. Rev 2021 · OpenReview

TL;DR

This review paper provides a comprehensive overview of vision-based robotic grasping, breaking it down into three core tasks: object localization, pose estimation, and grasp estimation. It meticulously surveys traditional and deep learning methods, highlighting their strengths, weaknesses, and future directions, offering a valuable resource for researchers in the field.

What Problem Does This Paper Address?

Robotic grasping, a fundamental capability for intelligent robots to interact with the physical world, faces significant challenges. While industrial robots can perform pick-and-place tasks, enabling robots to grasp diverse objects in complex, real-world environments remains a difficult problem. This paper addresses the need for a structured understanding of the vision-based approaches that enable robots to perceive and successfully grasp objects. It aims to provide a comprehensive survey of the state-of-the-art in object localization, pose estimation, and grasp estimation, which are crucial for achieving robust robotic grasping.

Methods

The paper categorizes vision-based robotic grasping methods into three main tasks:

  • Object Localization: This involves identifying the target object's location. It's further divided into:
  • Localization without classification: Finding object regions without knowing their category.
  • Object detection: Identifying object bounding boxes and their categories.
  • Object instance segmentation: Precisely segmenting individual object instances at the pixel or point level. Both 2D (RGB images) and 3D (point clouds, depth images) approaches are discussed.

  • Object Pose Estimation: This task estimates the 6D pose (position and orientation) of an object. The paper reviews three primary approaches:

  • Correspondence-based methods: Finding correspondences between observed data and a 3D object model.
  • Template-based methods: Matching observed data against a library of pre-defined templates with known poses.
  • Voting-based methods: Pixels or points vote for potential poses, which are then aggregated. Both traditional and deep learning techniques are covered for 2D and 3D data.

  • Grasp Estimation: This focuses on determining the gripper's pose for a successful grasp. It's divided into:

  • 2D planar grasp: Grasps constrained to a single direction, often represented by oriented rectangles or contact points.
  • 6DoF grasp: Grasps that can be executed from any angle in 3D space. Methods are discussed based on whether they use partial point clouds or complete object shapes.

The review emphasizes the increasing reliance on deep learning methods, particularly Convolutional Neural Networks (CNNs), for all these tasks, noting their superior performance in complex environments.

Datasets

The paper references several datasets commonly used in robotic grasping research, including:

  • LineMod: A benchmark for 6D object pose estimation.
  • IC-MI/IC-BIN: Datasets for 6D pose estimation.
  • T-LESS: Another dataset for 6D pose estimation of texture-less objects.
  • RU-APC: Dataset for robotic grasping.
  • YCB-Video: A widely used dataset for 6D pose estimation.
  • Cornell Grasping dataset: A popular dataset for 2D planar grasp estimation.
  • Stanford Grasping dataset: Used for 2D planar grasp datasets.
  • CMU dataset: Used for 2D planar grasp datasets.
  • Dex-Net 2.0: A dataset for robotic grasping.
  • JACQUARD dataset: Used for 2D planar grasp experiments.

The paper does not introduce new datasets but discusses how existing ones are used for evaluation.

Key Insights and Ideas

  • Task Decomposition is Key: The paper effectively breaks down the complex problem of robotic grasping into three manageable, sequential tasks: localization, pose estimation, and grasp estimation. This structured approach is crucial for understanding the field.
  • Deep Learning Dominance: Deep learning, especially CNNs, has revolutionized all aspects of robotic grasping, significantly improving performance in challenging scenarios compared to traditional methods.
  • RGB-D Data is Crucial: The advent of RGB-D sensors has been a game-changer, providing rich 3D information that is essential for robust pose and grasp estimation, especially for 6DoF grasps.
  • The "Grasp Estimation" is the Ultimate Goal: While localization and pose estimation are vital prerequisites, the ultimate objective is to determine a successful grasp pose, which is directly addressed by grasp estimation methods.
  • Challenges Remain: Despite significant progress, challenges like grasping in cluttered scenes, handling occlusions, the need for large annotated datasets, and grasping transparent objects persist, pointing towards exciting future research avenues.

Key Results

The paper is a survey, so it doesn't present new experimental results. Instead, it synthesizes the results of numerous studies. Key findings from the reviewed literature include:

  • High Accuracy Achieved: State-of-the-art methods, particularly deep learning-based ones, achieve high accuracies on benchmark datasets for object detection, pose estimation, and grasp estimation. For instance, on the YCB-Video dataset, methods like MaskedFusion and DenseFusion show AUC scores in the low 90s.
  • Speed vs. Accuracy Trade-off: One-stage detection methods generally offer higher speeds, while two-stage methods often provide better accuracy.
  • Robustness in Clutter is Difficult: While methods perform well on clean datasets, performance degrades significantly in cluttered environments with occlusions.
  • Category-Level Grasping is Emerging: Recent work is moving towards category-level grasp detection, which is more generalizable than instance-level approaches.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Insufficient Information in Data Acquisition: Current methods often rely on single-view RGB-D data, lacking information about the object's backside. Future work should focus on multi-view data fusion and incorporating multi-sensor data (e.g., haptic information).
  • Insufficient Training Data: The scarcity of large, diverse datasets hinders the development of generalizable models. Future directions include simulated environments for data generation and domain adaptation techniques, as well as semi-supervised and self-supervised learning.
  • Generalities in Grasping Novel Objects: Current methods struggle with objects not seen during training. Future work should focus on category-level grasp detection and incorporating more semantic information (e.g., object material, weight).
  • Grasping Transparent Objects: This remains a significant challenge due to the difficulty in capturing their 3D information. Future research should continue to explore methods for accurate 3D geometry reconstruction of transparent objects.
  • Cluttered Environments and Occlusions: Handling complex scenes with occlusions is a persistent challenge. Future work should focus on developing more robust methods for pose estimation and grasp detection in cluttered scenes.

Summary updated

Abstract

This paper presents a comprehensive survey on vision-based robotic grasping. We conclude three key tasks during vision-based robotic grasping, which are object localization, object pose estimation and grasp estimation. In detail, the object localization task contains object localization without classification, object detection and object instance segmentation. This task provides the regions of the target object in the input data. The object pose estimation task mainly refers to estimating the 6D object pose and includes correspondence-based methods, template-based methods and voting-based methods, which affords the generation of grasp poses for known objects. The grasp estimation task includes 2D planar grasp methods and 6DoF grasp methods, where the former is constrained to grasp from one direction. These three tasks could accomplish the robotic grasping with different combinations. Lots of object pose estimation methods need not object localization, and they conduct object localization and object pose estimation jointly. Lots of grasp estimation methods need not object localization and object pose estimation, and they conduct grasp estimation in an end-to-end manner. Both traditional methods and latest deep learning-based methods based on the RGB-D image inputs are reviewed elaborately in this survey. Related datasets and comparisons between state-of-the-art methods are summarized as well. In addition, challenges about vision-based robotic grasping and future directions in addressing these challenges are also pointed out.
BibTeX
@inproceedings{du2021vision,
  author = {Guoguang Du and Kai Wang and Shiguo Lian and Kaiyong Zhao},
  title = {{Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review}},
  booktitle = {Artif. Intell. Rev},
  year = {2021},
  url = {https://openreview.net/forum?id=qzFhqfuEBw0},
}
Figure