Inpaint2Learn: A Self-Supervised Framework for Affordance Learning

Lingzhi Zhang, Weiyu Du, Shenghao Zhou, Jiancong Wang, Jianbo ShiWACV 2022

TL;DR

This paper introduces Inpaint2Learn, a novel self-supervised framework that automatically generates affordance labels for images, enabling affordance learning in diverse, "in-the-wild" scenarios. By leveraging image inpainting and existing off-the-shelf prediction tools, Inpaint2Learn creates large-scale datasets for tasks like human pose prediction, object insertion, and 6D object pose estimation, demonstrating impressive results and outperforming previous methods.

What Problem Does This Paper Address?

Understanding "affordances" – the opportunities for interaction an environment offers – is crucial for both humans and AI systems to effectively perceive and interact with the world. However, acquiring large-scale, well-labeled datasets for affordance learning is a significant challenge. Existing methods often rely on limited, curated datasets (like sitcom videos) which struggle to generalize to the vast diversity of real-world scenes and interactions. This paper aims to bridge this gap by proposing a method to automatically generate affordance labels from any RGB image, thus enabling affordance learning in unconstrained, "in-the-wild" settings.

Methods

The core of Inpaint2Learn is a data preprocessing pipeline that generates affordance labels without requiring manual annotation. For a given RGB image, the pipeline first identifies an object of interest (e.g., a person, a car) using instance segmentation. This object is then "cut out," and the resulting hole in the image is filled using an image inpainting technique. The inpainted image serves as the "context" input for downstream models, while the cut-out object and its original location provide the pseudo-label for affordance prediction.

The framework is applied to three specific tasks:

  • Human Affordance Prediction: A two-stage pipeline is used. First, a conditional VAE predicts plausible human bounding boxes (location and shape). Second, another conditional VAE, conditioned on the predicted bounding boxes and auxiliary information (semantic segmentation and depth), synthesizes plausible human poses. A key innovation here is a "semantics-and-geometry-aware adversarial learning strategy" that uses a discriminator to ensure predictions are both semantically meaningful and geometrically consistent with the scene.
  • Location2Object: This task predicts what object to insert at a user-specified location and its bounding box size. A two-stage network takes a Gaussian mask of the location, RGB image, depth map, and semantic segmentation map as input. It first predicts the object's semantic class and then its bounding box.
  • 6D Object Pose Hallucination: This task aims to predict the 3D translation and rotation required to place a given object into a scene. A two-stage regression network, using ResNet-18 as a backbone, first predicts translation and then estimates rotation. The use of quaternions for rotation and the incorporation of depth information are highlighted as crucial for performance.

Datasets

The paper does not introduce entirely new datasets but rather leverages existing ones and generates new pseudo-labeled datasets using the Inpaint2Learn pipeline:

  • Human Affordance Prediction: The MPII dataset is used for evaluation.
  • Location2Object: The Cityscapes dataset is used. The authors generated 222,844 training data points and 10,000 test data points.
  • 6D Object Pose Hallucination: The Linemod dataset is used.

The authors plan to release the generated affordance label datasets for all three tasks.

Key Insights and Ideas

  • Self-supervised affordance learning is feasible: The core idea that you can generate meaningful affordance labels automatically from existing images by combining segmentation and inpainting is a significant contribution. This bypasses the need for expensive manual annotation.
  • "In-the-wild" affordance learning is possible: By using diverse datasets and off-the-shelf tools, the framework can learn affordances in unconstrained environments, overcoming the limitations of curated datasets.
  • Semantics and geometry are crucial for human pose prediction: The paper emphasizes that combining semantic segmentation and depth information significantly improves the plausibility and geometric correctness of predicted human poses.
  • Adversarial learning for plausibility: The use of a semantics-and-geometry-aware discriminator is key to ensuring that the generated predictions are not only accurate but also visually and contextually reasonable.
  • Quaternions and depth improve 6D pose estimation: For 6D object pose hallucination, using quaternions for rotation and incorporating depth data leads to substantial improvements in accuracy.

Key Results

  • Human Affordance Prediction: The full model (RGB + Depth + Segmentation) achieved the highest BBox Score (70.75) and Pose Score (75.94) in user studies, outperforming simpler baselines. The diversity of predictions was also on par with other models.
  • Location2Object: The full model achieved the highest classification accuracy (95.51%) and the best Semantic Score (73.16%) and Shape Score (79.78%) in user studies, demonstrating its effectiveness in predicting object classes and bounding boxes.
  • 6D Object Pose Hallucination: Using quaternions and depth information resulted in a significant boost in user study satisfaction rates, from 13.6170% (Euler angles) to 52.3404% (Quaternion) and 59.5745% (Quaternion & Depth). Translation and rotation errors were also minimized with the full model.

Limitations and Future Work

  • Reliance on off-the-shelf tools: The performance of Inpaint2Learn is inherently tied to the quality of the underlying instance segmentation, inpainting, and pose estimation tools. Errors in these components can propagate.
  • Video domain extension: The authors suggest expanding human affordance learning to the video domain, leveraging video inpainting techniques to study dynamic interactions and affordances in motion.
  • Generalization to unseen object types: While the framework is task-agnostic, its ability to generalize to entirely novel object categories not present in the training data for the underlying tools might be limited.

Summary updated

BibTeX
@inproceedings{zhang2022inpaint,
  author = {Lingzhi Zhang and Weiyu Du and Shenghao Zhou and Jiancong Wang and Jianbo Shi},
  title = {{Inpaint2Learn: A Self-Supervised Framework for Affordance Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure