CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images

Sookwan Han, Hanbyul Joo

Abstract

CHORUS (Canonicalized 3D Human-Object Spatial Relations) is a self-supervised framework that teaches machines 3D spatial "common sense" by leveraging generative AI. By synthesizing an "unbounded" dataset of human-object interactions (HOI) from diverse viewpoints and mapping them into a canonical 3D space, the method learns where objects are likely to be located relative to the human body without requiring any manual 3D annotations.

Introduction

Understanding how humans interact with objects in 3D—knowing that a hat goes on a head or a surfboard is ridden under the feet—is a fundamental challenge for robotics, augmented reality, and computer vision. The primary bottleneck is data: manually annotating 3D spatial relationships is incredibly difficult and does not scale. While the internet provides many 2D photos, these are often biased by professional photography angles and lack viewpoint diversity. CHORUS addresses this by learning robust 3D HOI priors from inconsistent 2D cues in a completely self-supervised manner.

Methodology

The CHORUS pipeline consists of four main stages designed to transform 2D generative outputs into structured 3D knowledge:

  1. Controllable Data Synthesis: The system uses ChatGPT to generate diverse interaction prompts (e.g., "A person carries a surfboard") and Stable Diffusion for image generation. Crucially, "viewpoint augmentation" (e.g., adding "+ top view") ensures the model produces a multi-view dataset of the same interaction.
  2. Filtering and 3D Estimation: Synthetic images are filtered for quality. For each valid image, the system estimates the 3D human pose using the SMPL model, which serves as a "3D anchor."
  3. Pose Canonicalization: To aggregate information from thousands of images, CHORUS warps 2D object masks from various poses into a single "canonical" 3D rest pose. This allows the model to perform 3D occupancy reasoning—essentially "voting" on which 3D voxels are most likely occupied by the object relative to the human.
  4. Semantic Clustering: Since different interactions (e.g., "riding" vs. "carrying") result in different spatial arrangements, the model uses text prompts and contact-point heuristics to cluster interactions into distinct 3D distributions.
The CHORUS pipeline: from auto-generated prompts to learned 3D spatial relations.

Key Insights and Results

A striking finding of this research is that generative models, despite occasional visual artifacts, are superior to internet search for learning 3D relations. They provide better viewpoint control and a more diverse range of "long-tail" interactions. The authors demonstrate that viewpoint diversity is more valuable than image realism for 3D spatial reasoning; a "surround-view" of a synthetic interaction provides more information than high-quality but biased real-world images.

By using the SMPL parametric human model as a coordinate system, the researchers turned a chaotic 2D alignment problem into a structured 3D aggregation problem. This allows the model to learn, for instance, that a "bicycle" distribution should look like a donut shape when a person is standing next to it, regardless of the camera angle.

Canonicalized human-object spatial distributions showing pose and semantic variations.

Quantitatively, the authors introduced the Projective Average Precision (PAP) metric. CHORUS trained on synthetic data consistently outperformed models trained on internet-crawled images (e.g., 57.14 mPAP for synthetic motorcycles vs. 55.97 for image search). Ablation studies confirmed that removing "pose canonicalization" caused a significant drop in performance, proving that aligning data to a common 3D reference frame is essential.

Conclusion and Future Work

CHORUS demonstrates a path toward learning 3D spatial priors without manual labels. While currently limited by a low-resolution voxel grid (48³) and challenges with small objects, the framework shows that generative AI can act as an "unbounded" teacher for 3D common sense. Future directions include moving toward higher-resolution representations like NeRF-based volumes and incorporating open-vocabulary object detectors to expand the system's interactable object vocabulary.

Summary updated

Abstract

We present a method for teaching machines to understand and model the underlying spatial common sense of diverse human-object interactions in 3D in a self-supervised way. This is a challenging task, as there exist specific manifolds of the interactions that can be considered human-like and natural, but the human pose and the geometry of objects can vary even for similar interactions. Such diversity makes the annotating task of 3D interactions difficult and hard to scale, which limits the potential to reason about that in a supervised way. One way of learning the 3D spatial relationship between humans and objects during interaction is by showing multiple 2D images captured from different viewpoints when humans interact with the same type of objects. The core idea of our method is to leverage a generative model that produces high-quality 2D images from an arbitrary text prompt input as an "unbounded" data generator with effective controllability and view diversity. Despite its imperfection of the image quality over real images, we demonstrate that the synthesized images are sufficient to learn the 3D human-object spatial relations. We present multiple strategies to leverage the synthesized images, including (1) the first method to leverage a generative image model for 3D human-object spatial relation learning; (2) a framework to reason about the 3D spatial relations from inconsistent 2D cues in a self-supervised manner via 3D occupancy reasoning with pose canonicalization; (3) semantic clustering to disambiguate different types of interactions with the same object types; and (4) a novel metric to assess the quality of 3D spatial learning of interaction.
BibTeX
@inproceedings{han2023chorus,
  author = {Sookwan Han and Hanbyul Joo},
  title = {{CHORUS: Learning Canonicalized 3D Human-Object Spatial Relations from Unbounded Synthesized Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure