Monocular Human-Object Reconstruction in the Wild

Chaofan Huo, Ye Shi, Jingya Wang

Abstract

Reconstructing 3D human-object interactions (HOI) from a single image is a fundamental challenge in computer vision. Traditional methods rely on scarce 3D-annotated datasets or rigid rule-based priors. This paper introduces a novel 2D-supervised approach that leverages the vast diversity of Internet images to learn spatial priors of human-object relations. By treating the Internet as an unorganized multi-view camera system, the proposed method achieves performance comparable to 3D-supervised models while demonstrating superior generalization in real-world scenarios.

Introduction

The ability to reconstruct 3D human-object interactions from monocular images is critical for applications in augmented reality, robotics, and human-computer interaction. However, this task is plagued by depth ambiguity and frequent occlusions. Existing approaches typically fall into two categories: 3D-supervised models trained on limited laboratory data (e.g., BEHAVE) and rule-based models that use symbolic "commonsense" constraints. The former often fails to generalize to the "wild," while the latter lacks the precision needed for complex interactions.

Conceptual overview of learning 3D spatial priors from 2D Internet images.

The authors propose a data-driven alternative: learning flexible 3D spatial priors directly from 2D images. By grouping similar interactions across different images, the method effectively simulates a multi-view setup to resolve geometric ambiguities without requiring explicit 3D annotations.

Methodology

The proposed framework consists of two main stages: prior learning and post-optimization.

Prior Learning from 2D Images

The core of the method is a conditional Normalizing Flow model that learns the probability distribution of 2D human-object keypoint layouts ($X_{2.5D}$) given an input image $I$. To supervise this model without 3D data, the authors employ a K-Nearest Neighbor (KNN) grouping strategy. Images depicting similar interactions are clustered, providing the model with multiple "views" of the same interaction type. This allows the network to capture the underlying 3D structure through 2D consistency.

The two-stage pipeline: (a) Prior learning using normalizing flows and KNN grouping, and (b) Post-optimization for 3D reconstruction.

Post-Optimization with 3D Priors

During the reconstruction phase, initial 3D poses for the human (SMPLer-X) and the object (CDPN) are estimated. These poses are then refined by maximizing their likelihood under the learned 2D prior. This optimization is performed across multiple "virtual" camera views to ensure 3D coherence. Additionally, a contact loss is derived from mean occlusion maps, which are computed by averaging human-object masks across the dataset. This allows the model to infer contact points without manual labels.

Datasets

The study introduces WildHOI, a new large-scale benchmark collected from YouTube. It features 8 object categories (e.g., baseball bat, skateboard, cello) in diverse real-world settings, containing over 30,000 training frames and a manually annotated test set. Comparisons are also conducted on the BEHAVE dataset, a standard indoor benchmark with full 3D ground truth.

Experimental Results

The 2D-supervised method demonstrates remarkable effectiveness:

  • Indoor Performance: On the BEHAVE dataset, the method achieved a human mesh error of 4.55cm, nearly matching the 4.34cm of fully 3D-supervised models.
  • In-the-Wild Generalization: On the WildHOI dataset, the proposed method significantly outperformed the previous state-of-the-art (PHOSA), reducing object translation error from 33.22cm to 13.13cm.
  • Qualitative Quality: Reconstructions are physically plausible and well-grounded, even in cases where the human is not in direct contact with the object (e.g., mid-air skateboarding).

Conclusion

This work demonstrates that 2D Internet images provide a rich source of geometric information for learning 3D human-object interaction priors. By moving away from the requirement for expensive 3D annotations, the proposed method paves the way for more robust and scalable 3D reconstruction in unconstrained environments. Future work will explore category-level priors and the reconstruction of deformable objects.

Summary updated

Abstract

Learning the prior knowledge of the 3D human-object spatial relation is crucial for reconstructing human-object interaction from images and understanding how humans interact with objects in 3D space. Previous works learn this prior from datasets collected in controlled environments, but due to the diversity of domains, they struggle to generalize to real-world scenarios. To overcome this limitation, we present a 2D-supervised method that learns the 3D human-object spatial relation prior purely from 2D images in the wild. Our method utilizes a flow-based neural network to learn the prior distribution of the 2D human-object keypoint layout and viewports for each image in the dataset. The effectiveness of the prior learned from 2D images is demonstrated on the human-object reconstruction task by applying the prior to tune the relative pose between the human and the object during the post-optimization stage. To validate and benchmark our method on in-the-wild images, we collect the WildHOI dataset from the YouTube website, which consists of various interactions with 8 objects in real-world scenarios. We conduct the experiments on the indoor BEHAVE dataset and the outdoor WildHOI dataset. The results show that our method achieves almost comparable performance with fully 3D supervised methods on the BEHAVE dataset, even if we have only utilized the 2D layout information, and outperforms previous methods in terms of generality and interaction diversity on in-the-wild images.
BibTeX
@inproceedings{huo2024monocular,
  author = {Chaofan Huo and Ye Shi and Jingya Wang},
  title = {{Monocular Human-Object Reconstruction in the Wild}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681452},
}
Figure