HOI-M3: Capture Multiple Humans and Objects Interaction within Contextual Environment

Juze Zhang, Jingyan Zhang, Zining Song, Zhanhe Shi, Chengfeng Zhao, Ye Shi, Jingyi Yu, Lan Xu, Jingya WangCVPR 2024 · DOI · OpenReview

Abstract

Human-Object Interaction (HOI) research has traditionally focused on isolated individuals and single objects. This paper introduces HOI-M3, a large-scale dataset and benchmark designed to scale HOI to complex social scenes involving multiple humans and multiple objects. By utilizing a hybrid capture system combining 4K camera arrays with object-mounted IMUs, the authors address the challenge of severe occlusion in multi-person environments. We present a monocular baseline for 3D pose estimation and a generative model for synthesizing social interactions.

Introduction

Human-Object Interaction (HOI) is a cornerstone of computer vision, essential for applications ranging from AR/VR to robotics. However, most existing research and datasets are limited to a "single person, single object" paradigm. Real-world scenarios are rarely that simple; they involve complex social interactions such as cooking together, working out in a group, or sharing a meal.

HOI-M3 captures complex social interactions involving multiple humans and objects across various indoor environments.

Modeling these scenarios is incredibly difficult due to severe occlusion: when multiple people interact with multiple objects, they constantly block the cameras' line of sight. This data scarcity has prevented the development of robust models capable of understanding or synthesizing complex social behaviors in contextual environments. HOI-M3 addresses this by providing a massive, high-quality dataset of multi-person social behaviors.

Methods

To solve the occlusion problem, the authors developed a hybrid capture system. They utilized a dome-like setup with 42 synchronized 4K cinema cameras for dense visual coverage. To track objects even when they are completely hidden by a person’s body, they embedded Inertial Measurement Units (IMUs) directly into the objects.

The technical pipeline consists of several key stages:

  1. Human Capture: Utilizing ViTPose for 2D keypoints, triangulated into 3D, and fitted to the SMPL (Skinned Multi-Person Linear) body model.
  2. Inertial-aided Object Tracking: Object poses are optimized using a combination of visual masks (from the Segment Anything Model) and IMU rotation data, including constraints for collisions and smoothness.
  3. Monocular Baseline: A one-stage deep learning pipeline that estimates the 3D poses of multiple people and objects from a single RGB image using a unified heatmap approach.
The proposed monocular baseline architecture utilizes a unified heatmap approach and parameter sampling to estimate 3D poses for multiple humans and objects.

The HOI-M3 Dataset

The core contribution is the HOI-M3 dataset, which is significantly larger and more complex than its predecessors:

  • Scale: 181 million video frames across 199 sequences.
  • Diversity: 31 human subjects and 90 different 3D objects.
  • Complexity: Every sequence involves at least 2 persons and 5 objects.
  • Scenarios: Covers five daily environments: Bedroom, Dining Room, Living Room, Fitness Room, and Office.

Compared to previous datasets like GTA-IM (synthetic) or PROX (static scenes), HOI-M3 is the first to offer real-world, dynamic, multi-person, and multi-object tracking with high-resolution ground truth.

Key Results

The authors benchmarked their monocular capture method against state-of-the-art (SOTA) single-HOI models like PHOSA and CHORE.

  • Accuracy: In multi-person scenarios, the HOI-M3 baseline achieved a PCK (Percentage of Correct Keypoints) of 68.5%, vastly outperforming CHORE (10.4%) and PHOSA (43.9%), which struggle when the scene becomes crowded.
  • Object Tracking: Their method reduced the "Vertex-to-Vertex" (V2V) error for objects to 297.8mm, compared to over 690mm for PHOSA.
  • Generation: Their diffusion-based model successfully synthesized semantically meaningful motions (e.g., two people sitting at a table together) that respected the physical boundaries of the objects.

Limitations and Future Work

While a major step forward, the authors note several constraints. The capture is currently limited to indoor lab settings with controlled illumination. Expanding this to "in-the-wild" outdoor environments remains a significant challenge due to the hardware requirements of 42 cameras and IMU-embedded objects. Future work will likely focus on moving these capabilities to unstructured environments and improving the "common sense" physics of generated interactions.

Summary updated

Abstract

Humans naturally interact with both others and the surrounding multiple objects, engaging in various social activities. However, recent advances in modeling human-object interactions mostly focus on perceiving isolated individuals and objects, due to fundamental data scarcity. In this paper, we introduce HOI-M3, a novel large-scale dataset for modeling the interactions of Multiple huMans and Multiple objects. Notably, it provides accurate 3D tracking for both humans and objects from dense RGB and object-mounted IMU inputs, covering 199 sequences and 181M frames of diverse humans and objects under rich activities. With the unique HOI-M3 dataset, we introduce two novel data-driven tasks with companion strong baselines: monocular capture and unstructured generation of multiple human-object interactions. Extensive experiments demonstrate that our dataset is challenging and worthy of further research about multiple human-object interactions and behavior analysis. Our HOI-M3 dataset, corresponding codes, and pre-trained models will be disseminated to the community for future research.
BibTeX
@inproceedings{zhang2024hoi,
  author = {Juze Zhang and Jingyan Zhang and Zining Song and Zhanhe Shi and Chengfeng Zhao and Ye Shi and Jingyi Yu and Lan Xu and Jingya Wang},
  title = {{HOI-M3: Capture Multiple Humans and Objects Interaction within Contextual Environment}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00056},
}
Figure