Abstract
CHORUS (Canonicalized 3D Human-Object Spatial Relations) is a self-supervised framework that teaches machines 3D spatial "common sense" by leveraging generative AI. By synthesizing an "unbounded" dataset of human-object interactions (HOI) from diverse viewpoints and mapping them into a canonical 3D space, the method learns where objects are likely to be located relative to the human body without requiring any manual 3D annotations.
Introduction
Understanding how humans interact with objects in 3D—knowing that a hat goes on a head or a surfboard is ridden under the feet—is a fundamental challenge for robotics, augmented reality, and computer vision. The primary bottleneck is data: manually annotating 3D spatial relationships is incredibly difficult and does not scale. While the internet provides many 2D photos, these are often biased by professional photography angles and lack viewpoint diversity. CHORUS addresses this by learning robust 3D HOI priors from inconsistent 2D cues in a completely self-supervised manner.
Methodology
The CHORUS pipeline consists of four main stages designed to transform 2D generative outputs into structured 3D knowledge:
- Controllable Data Synthesis: The system uses ChatGPT to generate diverse interaction prompts (e.g., "A person carries a surfboard") and Stable Diffusion for image generation. Crucially, "viewpoint augmentation" (e.g., adding "+ top view") ensures the model produces a multi-view dataset of the same interaction.
- Filtering and 3D Estimation: Synthetic images are filtered for quality. For each valid image, the system estimates the 3D human pose using the SMPL model, which serves as a "3D anchor."
- Pose Canonicalization: To aggregate information from thousands of images, CHORUS warps 2D object masks from various poses into a single "canonical" 3D rest pose. This allows the model to perform 3D occupancy reasoning—essentially "voting" on which 3D voxels are most likely occupied by the object relative to the human.
- Semantic Clustering: Since different interactions (e.g., "riding" vs. "carrying") result in different spatial arrangements, the model uses text prompts and contact-point heuristics to cluster interactions into distinct 3D distributions.
Key Insights and Results
A striking finding of this research is that generative models, despite occasional visual artifacts, are superior to internet search for learning 3D relations. They provide better viewpoint control and a more diverse range of "long-tail" interactions. The authors demonstrate that viewpoint diversity is more valuable than image realism for 3D spatial reasoning; a "surround-view" of a synthetic interaction provides more information than high-quality but biased real-world images.
By using the SMPL parametric human model as a coordinate system, the researchers turned a chaotic 2D alignment problem into a structured 3D aggregation problem. This allows the model to learn, for instance, that a "bicycle" distribution should look like a donut shape when a person is standing next to it, regardless of the camera angle.
Quantitatively, the authors introduced the Projective Average Precision (PAP) metric. CHORUS trained on synthetic data consistently outperformed models trained on internet-crawled images (e.g., 57.14 mPAP for synthetic motorcycles vs. 55.97 for image search). Ablation studies confirmed that removing "pose canonicalization" caused a significant drop in performance, proving that aligning data to a common 3D reference frame is essential.
Conclusion and Future Work
CHORUS demonstrates a path toward learning 3D spatial priors without manual labels. While currently limited by a low-resolution voxel grid (48³) and challenges with small objects, the framework shows that generative AI can act as an "unbounded" teacher for 3D common sense. Future directions include moving toward higher-resolution representations like NeRF-based volumes and incorporating open-vocabulary object detectors to expand the system's interactable object vocabulary.
Summary updated