Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance

Wenkai Chen, Hongzhuo Liang, Zhaopeng Chen, Fuchun Sun, Jianwei Zhang

Abstract

This paper presents a novel framework for 6-DoF task-oriented grasping that reasons directly on 3D point clouds. By integrating an Implicit Estimation Network (IEN) for generating diverse grasp candidates with a Visual Affordance Network (VAN) for 3D heatmaps, the system determines optimal grasp poses for specific tasks such as pouring, cutting, or passing. Experimental results demonstrate that the implicit modeling approach significantly outperforms traditional generative models in capturing complex grasp distributions.

Introduction

Standard robotic grasping typically focuses on "any-grasp" detection—ensuring an object is picked up securely without dropping it. However, real-world utility requires task-oriented grasping, where the robot must hold an object in a manner that facilitates a specific subsequent action. For instance, a mug should be grasped by the handle for pouring, rather than the rim.

Existing task-oriented methods often rely on 2D image segmentation to identify "affordance" regions. These methods face two primary challenges: the noise associated with mapping 2D pixels to 3D space and the difficulty of representing multi-modal grasp distributions (e.g., multiple ways to hold a handle) using standard generative models like Variational Autoencoders (VAEs).

Methodology

The proposed framework utilizes a two-stream architecture to process partial 3D point clouds:

  1. Implicit Estimation Network (IEN): Utilizing Implicit Maximum Likelihood Estimation (IMLE), this network generates diverse 6-DoF grasp candidates. Unlike VAEs, which often suffer from mode collapse, the implicit nature of this network allows it to capture complex, non-Gaussian distributions of valid task-specific grasps.
  2. Visual Affordance Network (VAN): This module predicts a 3D heatmap across the object's surface. It employs an attention-aware bilinear feature learning approach, combining PointNet and DGCNN architectures with Multi-Head Self-Attention (MHSA) to synthesize local geometry and global semantic context.
  3. Grasp Refinement: The 3D affordance heatmap serves as a guide to filter and refine the coarse candidates from the IEN, ensuring the final grasp poses are centered on task-relevant regions.
Figure 1: The two-stream framework architecture, showing the parallel processing of the Implicit Estimation Network and the Visual Affordance Network to produce refined task-oriented grasps.

Datasets and Evaluation

To address the lack of 6-DoF task-oriented data, the authors developed the Affordance Grasp Dataset. Built upon ACRONYM and ShapeNet, it contains 203 household objects across six categories (Mug, Bottle, Knife, Hat, Bowl, Scissor) with over 100,000 annotated successful grasps for six common tasks. The visual affordance module was trained using the 3D AffordanceNet dataset.

Performance was measured using the Evaluated Similarity Metric (ESM), which quantifies the distance between predicted grasps and ground truth. The IEN model achieved an ESM of 0.092, a significant improvement over the VAE baseline of 0.151 (where lower is better).

Key Insights and Results

A critical finding is that implicit models are superior to explicit generative models for 3D grasp detection. VAEs tend to over-smooth complex 3D spaces, whereas implicit models allow the robot to discover distinct clusters of valid grasps for a single task. Furthermore, using heatmaps as soft guidance rather than hard masks makes the system more robust to sensor noise.

Figure 2: Visualizations of task-oriented grasp distributions for various objects. The clusters demonstrate how the model identifies specific functional regions like handles for different tasks.

The model demonstrated strong generalization to novel objects and was successfully validated in real-world experiments using a KUKA LWR robot. It accurately executed task-specific grasps on physical objects, such as 3D-printed knives and mugs, even under random placement conditions.

Conclusion

While the framework shows promise, limitations remain regarding sensitivity to smooth or irregular textures and the current diversity of object categories. Future work will focus on expanding the dataset and exploring complex human-robot interaction tasks, such as hand-overs, where affordance must account for the receiver's ease of use.

Summary updated

BibTeX
@inproceedings{chen2022learning,
  author = {Wenkai Chen and Hongzhuo Liang and Zhaopeng Chen and Fuchun Sun and Jianwei Zhang},
  title = {{Learning 6-DoF Task-oriented Grasp Detection via Implicit Estimation and Visual Affordance}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  doi = {10.1109/iros47612.2022.9981900},
}
Figure