Attention based visual analysis for fast grasp planning with multi-fingered robotic hand

Zhen Deng, Ge Gao, Simone Frintrop, Jianwei ZhangarXiv 2018 · arXiv

Abstract

Grasping objects with multi-fingered robotic hands presents a high-dimensional challenge due to the vast search space of possible configurations. This paper introduces an attention-based visual framework that accelerates grasp planning by integrating a bottom-up saliency model with a top-down deep neural network. The system identifies regions of interest and predicts specific grasp types from a six-category taxonomy. Experimental results demonstrate an 88.3% success rate and an 80% reduction in search attempts compared to traditional methods, showcasing robust generalization across different robotic platforms and unknown objects.

Introduction: The Challenge of Multi-Fingered Grasping

Grasping objects with a multi-fingered robotic hand, such as the Shadow Hand, is a complex problem. Unlike simple two-fingered grippers, multi-fingered hands possess dozens of degrees of freedom, making the search for a stable grasp configuration computationally expensive. Existing approaches often suffer from heavy computational requirements or oversimplified grasp taxonomies (e.g., binary power vs. precision), which fail to capture the nuances of human-like interaction with complex geometries.

Proposed Methodology

The authors propose a dual-process framework inspired by human visual attention, combining bottom-up and top-down analysis to streamline the grasping process.

Overview of the attention-based visual analysis framework.

Bottom-Up: Saliency Detection

The system utilizes the VOCUS2 algorithm to generate pixel-precise saliency maps. By identifying regions of interest based on local contrast (color, intensity) without prior object knowledge, the robot can effectively target unknown objects in cluttered environments.

Top-Down: Grasp Prediction

Once a Region of Interest (ROI) is identified, a modified DeepLab Convolutional Neural Network predicts:

  1. Grasp Type: One of six categories: large wrap, small wrap, power, pinch, precision, and tripod.
  2. Grasp Point: The optimal pixel coordinates for the center of the grasp.

Grasp Planning

These predictions serve as prior information, allowing the planner to initialize a pre-grasp configuration and perform a localized search in a 4D space (offset and rotation angles), significantly narrowing the search space.

Experimental Setup and Datasets

The framework was trained and evaluated using several key resources:

  • Grasp Type Dataset (GTD): A custom dataset of 11,000 annotated images across 12 household objects, providing ground-truth labels for the six grasp types.
  • YCB Object Set: Used to test generalization to unseen objects.
  • Simulation: Testing was conducted in the V-REP physical simulator with models of the Shadow Hand, Barrett Hand, and Baxter gripper.

Key Insights

Saliency as a Computational Shortcut

By employing a handcrafted saliency model (VOCUS2) rather than a deep-learning-based alternative, the system avoids "photographer bias" and maintains high speed and hardware efficiency during initial localization.

Granular Taxonomy

Moving beyond binary classifications to a six-type taxonomy allows the robot to better match its hand shape to local object features. For instance, a "precision" grasp is more suitable for small, delicate interactions than a generic power grasp.

Example of a precision grasp from the six-type taxonomy.

Generalization to Unknown Objects

Because the framework focuses on local sub-regions and visual attributes rather than global object recognition, it successfully plans grasps for novel objects, such as bananas or specific soup cans, by identifying familiar local geometries.

Results

The proposed framework demonstrated significant improvements over baseline methods:

  • Accuracy: The grasp detection network achieved a 0.60 Mean Intersection-over-Union (IoU), a 10% improvement over SegNet-based approaches.
  • Efficiency: The system required only 3.5 search attempts on average to find a feasible grasp, compared to 20.5 attempts for random search.
  • Success Rate: An overall success rate of 88.3% was achieved, outperforming methods without attention-based priors (71.6%).
  • Hardware Agnostic: The framework generalized well to the 3-fingered Barrett hand (90% success) and the 2-fingered Baxter gripper (100% success).

Limitations and Future Work

Despite its success, the model has limitations. It currently does not explicitly account for environmental constraints like table surfaces, which can lead to collisions. Additionally, the processing time of ~29 seconds per frame requires further optimization for real-time applications. Future work will focus on making the framework "task-driven" and incorporating depth information for improved 3D pose estimation.

Summary updated

BibTeX
@misc{deng2018attention,
  author = {Zhen Deng and Ge Gao and Simone Frintrop and Jianwei Zhang},
  title = {{Attention based visual analysis for fast grasp planning with multi-fingered robotic hand}},
  year = {2018},
  eprint = {1809.04226},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.04226},
}
Figure