PartAfford: Part-level Affordance Discovery

Chao Xu, Yixin Chen, He Wang, Song-Chun Zhu, Yixin Zhu, Siyuan HuangICLR 2023 · OpenReview

TL;DR

This paper introduces "PartAfford," a novel task for discovering part-level object affordances from 3D shapes using only sparse, object-level supervision. They propose a new framework that leverages slot attention and geometric primitive regularization to jointly decompose objects into parts and predict their affordances, mimicking how humans learn.

What Problem Does This Paper Address?

Understanding how objects can be used (their affordances) is crucial for robots and AI to interact with the world. Traditional approaches often require dense, per-pixel supervision, which is unrealistic for how humans learn. Humans can infer affordances from limited information, often by understanding how different parts of an object contribute to its function. This paper tackles the challenge of learning object affordances in a more human-like way, specifically by learning to identify which part of an object affords what action, using only the affordance labels for the entire object, not for each individual part.

Methods

The authors propose a novel framework called PartAfford with two main components:

  • Abstraction Encoder: This component takes a 3D object (represented as voxels) and uses a 3D convolutional neural network to extract features. These features are then fed into a 3D Slot Attention module. Slot attention is a mechanism that learns to abstract the input into a set of "slots," where each slot can represent a distinct object part or feature. This is done in an unsupervised manner, aiming to disentangle the object's components.
  • Affordance Decoder: This decoder takes the learned slot representations and has three branches:
  • 3D Part Reconstruction: This branch reconstructs the 3D parts of the object from the slots. This is self-supervised, meaning the model tries to reconstruct the original object from these predicted parts.
  • Affordance Prediction: This branch predicts affordance labels for each slot. Crucially, this uses sparse set supervision, meaning it only receives the set of affordance labels for the entire object, not for individual parts. The model learns to align the predicted part affordances with the overall object affordances using a Hungarian matching algorithm.
  • Cuboidal Primitive Regularization: This branch encourages the predicted object parts to conform to cuboidal shapes. This geometric prior helps to regularize the segmentation process, especially for densely connected parts, and improves the discovery of affordance-bearing parts.

Datasets

The authors introduce a new dataset for training and evaluating PartAfford.

  • PartAfford Dataset: This dataset consists of over 25,000 3D CAD models annotated with 24 cross-category, fine-grained affordance labels for object parts. It is derived from the PartNet [32] and PartNet-Mobility [49] datasets. This dataset is unique in providing part-level affordance annotations with sparse, object-level supervision, differentiating it from prior work that often uses dense annotations.

Key Insights and Ideas

  • Sparse Supervision is Sufficient: The core idea is that humans don't need dense, per-part labels to learn affordances. This work demonstrates that even with just object-level affordance labels, a model can learn to decompose objects into meaningful parts and assign affordances to them.
  • Slot Attention for Part Discovery: The use of slot attention is key to unsupervised part discovery. By learning to attend to different regions of the 3D shape, slots can effectively disentangle object components.
  • Geometric Priors Aid Segmentation: Incorporating a cuboidal primitive regularization helps the model segment object parts more effectively, especially in complex scenes where parts might be densely connected or have ambiguous boundaries. This geometric constraint acts as a valuable prior.
  • Affordance Compositionality Matters: The authors highlight that affordances often co-occur (e.g., a chair has both "sittable" and "backrest" affordances). Understanding these compositions is important for accurate prediction.

Key Results

  • State-of-the-Art Performance: The proposed PartAfford method significantly outperforms baseline models on the PartAfford task, achieving higher mean IoU for part discovery and competitive affordance prediction accuracy.
  • Quantitative Improvements: For the "sittable" category, the full model achieved a mean IoU of 57.3%, significantly higher than baselines like Slot MLP (21.5%) and IODINE (31.5%).
  • Qualitative Success: The model successfully segments objects into parts and assigns plausible affordances, even for novel object shapes and unseen categories, demonstrating good generalization capabilities. Visualizations show how slots gradually learn to represent specific parts and their associated affordances.
  • Component Importance: Ablation studies show that each component (affordance prediction, cuboidal regularization, and data augmentation) contributes positively to the overall performance, with affordance prediction and data augmentation having a particularly strong impact on mean IoU.

Limitations and Future Work

  • Limited Affordance Categories: The study focused on only three representative affordances ("sittable," "support," "openable") due to the complexity of discovering other affordances (e.g., "rollable," "cutting") which might require richer interactions or visual cues beyond geometry.
  • Cuboid Assumption: The cuboidal primitive regularization might struggle with parts that significantly deviate from cuboid shapes.
  • Handle Affordance: The model had difficulty discovering and reconstructing "handle" affordances, possibly due to the diverse shapes and contexts in which handles appear.
  • Future Directions:
    • Visual Appearance: Incorporating visual appearance (texture, color) could further improve shape abstraction and affordance discovery.
  • Active Learning: Developing an active learning approach could allow the model to efficiently gather more informative samples, similar to how humans learn through interaction.
  • Multiple Affordances: Extending the framework to handle multiple affordances per part (currently uses one-hot encoding) is a promising avenue.
  • More Complex Affordances: Exploring affordances that require more than just geometric information (e.g., active interactions, material properties) is a key area for future research.

Summary updated

Abstract

Understanding what objects could furnish for humans—learning object affordance—is the crux of bridging perception and action. In the vision community, prior work has primarily focused on learning object affordance with dense (e.g., at a per-pixel level) supervision. In stark contrast, we humans learn the object affordance without dense labels. As such, the fundamental question to devise a computational model is: What is the natural way to learn the object affordance from geometry with humanlike sparse supervision? In this work, we present the new task of part-level affordance discovery (PartAfford): Given only the affordance labels for each object, the machine is tasked to (i) decompose 3D shapes into parts and (ii) discover how each part of the object corresponds to a certain affordance category. We propose a novel learning framework that discovers part-level representations by leveraging only the affordance set supervision and geometric primitive regularization without dense supervision. To learn and evaluate PartAfford, we construct a part-level, cross-category 3D object affordance dataset, annotated with 24 affordance categories shared among >25, 000 objects. We demonstrate through extensive experiments that our method enables both the abstraction of 3D objects and part-level affordance discovery, with generalizability to difficult and cross-category examples. Further ablations reveal the contribution of each component.
BibTeX
@inproceedings{xu2023partafford,
  author = {Chao Xu and Yixin Chen and He Wang and Song-Chun Zhu and Yixin Zhu and Siyuan Huang},
  title = {{PartAfford: Part-level Affordance Discovery}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=bQZ2wEYxRBL},
}
Figure