AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose

Juntao Jian, Xiuping Liu, Manyi Li, Ruizhen Hu, Jian LiuICCV 2023

Abstract

AffordPose is a large-scale dataset featuring 26.7K hand-object interactions designed to bridge the gap between physical grasping and functional usage. Unlike existing datasets that focus on general stability, AffordPose provides fine-grained, part-level affordance labels (e.g., "twist," "pull," "press") and corresponding 3D hand poses. This work enables AI agents to learn functional interactions by mapping detailed hand configurations to specific object parts.

Introduction

In robotics and computer vision, teaching an agent to simply "pick up a mug" is largely a solved problem. However, functional interaction—such as picking up a mug by the handle to drink versus grasping the rim to wash it—remains a significant challenge. Existing datasets often focus on general grasping for stability or high-level intents (e.g., "use," "handover") that are too vague for precise robotic manipulation.

AffordPose addresses this lack of fine-grained data by arguing that human hand poses are primarily driven by affordances—the functional roles of specific object parts. By mapping detailed 3D hand configurations to specific functional zones, the dataset provides a framework for learning how to interact with an object's functional components.

Methodology

The researchers developed a two-stage pipeline to construct the AffordPose dataset, ensuring both semantic accuracy and physical plausibility.

  1. Part-Level Affordance Labeling: Using 3D models from PartNet and PartNet-Mobility, objects were segmented into functional parts. Volunteers assigned one of eight specific affordance labels to these segments.
  2. Affordance-Driven Hand Pose Annotation: Volunteers adjusted a MANO hand model (a standard 3D articulated model with 16 joints) to interact with the labeled parts. To ensure these poses were not merely "floating," the GraspIt! simulator was used to perform force analysis, validating that the poses were physically stable.
The two-stage annotation pipeline: part-level labeling followed by hand pose annotation.

To validate the utility of this data, the authors developed AffordPoseNet, a generative model that predicts appropriate hand poses based on an input object and a specific affordance label.

The AffordPose Dataset

The primary contribution of this work is a dataset characterized by its scale and granularity:

  • Scale: 26.7K unique interactions.
  • Diversity: 641 3D objects across 13 categories, including bags, bottles, faucets, and laptops.
  • Granularity: 8 distinct hand-centered affordance types: twist, pull, handle-grasp, press, lift, support, wrap-grasp, and lever.
The eight distinct hand-centered affordance types defined in the AffordPose dataset.

Unlike datasets like HO-3D or DexYCB, which focus on video capture of general grasps, AffordPose provides a direct semantic link between an object part's function and the resulting 3D hand pose.

Key Insights and Results

Affordance Over Category

A critical finding of the study is that hand poses correlate more strongly with the affordance than the object category. For example, a "handle-grasp" on a pot is remarkably similar to a "handle-grasp" on a mug. This suggests that learning affordances allows for better generalization across different object types.

Performance Metrics

  • Affordance Understanding: Using hand poses to predict affordance classification reached 98.39% accuracy.
  • Part Localization: Models identified the specific part of an object being interacted with at an Intersection over Union (IoU) rate of 96.29%.
  • Pose Generation: AffordPoseNet significantly outperformed baselines like GrabNet. While baselines often generated generic grasps at the center of an object, AffordPoseNet successfully generated distinct, functional poses for different parts of the same object (e.g., twisting a cap vs. grasping the bottle body).

Conclusion and Future Work

AffordPose provides a vital link between semantic function and physical pose. While the current dataset is limited to static snapshots and single-hand interactions, it lays the groundwork for future research into dynamic affordances (the motion of twisting or pulling) and bi-manual tasks. Future iterations aim to automate the annotation process to scale the dataset further and improve accuracy for interactions with very small functional parts, such as zippers or thin levers.

Summary updated

BibTeX
@inproceedings{jian2023affordpose,
  author = {Juntao Jian and Xiuping Liu and Manyi Li and Ruizhen Hu and Jian Liu},
  title = {{AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure