Abstract
Human-Object Interaction (HOI) research has traditionally focused on isolated individuals and single objects. This paper introduces HOI-M3, a large-scale dataset and benchmark designed to scale HOI to complex social scenes involving multiple humans and multiple objects. By utilizing a hybrid capture system combining 4K camera arrays with object-mounted IMUs, the authors address the challenge of severe occlusion in multi-person environments. We present a monocular baseline for 3D pose estimation and a generative model for synthesizing social interactions.
Introduction
Human-Object Interaction (HOI) is a cornerstone of computer vision, essential for applications ranging from AR/VR to robotics. However, most existing research and datasets are limited to a "single person, single object" paradigm. Real-world scenarios are rarely that simple; they involve complex social interactions such as cooking together, working out in a group, or sharing a meal.
Modeling these scenarios is incredibly difficult due to severe occlusion: when multiple people interact with multiple objects, they constantly block the cameras' line of sight. This data scarcity has prevented the development of robust models capable of understanding or synthesizing complex social behaviors in contextual environments. HOI-M3 addresses this by providing a massive, high-quality dataset of multi-person social behaviors.
Methods
To solve the occlusion problem, the authors developed a hybrid capture system. They utilized a dome-like setup with 42 synchronized 4K cinema cameras for dense visual coverage. To track objects even when they are completely hidden by a person’s body, they embedded Inertial Measurement Units (IMUs) directly into the objects.
The technical pipeline consists of several key stages:
- Human Capture: Utilizing ViTPose for 2D keypoints, triangulated into 3D, and fitted to the SMPL (Skinned Multi-Person Linear) body model.
- Inertial-aided Object Tracking: Object poses are optimized using a combination of visual masks (from the Segment Anything Model) and IMU rotation data, including constraints for collisions and smoothness.
- Monocular Baseline: A one-stage deep learning pipeline that estimates the 3D poses of multiple people and objects from a single RGB image using a unified heatmap approach.
The HOI-M3 Dataset
The core contribution is the HOI-M3 dataset, which is significantly larger and more complex than its predecessors:
- Scale: 181 million video frames across 199 sequences.
- Diversity: 31 human subjects and 90 different 3D objects.
- Complexity: Every sequence involves at least 2 persons and 5 objects.
- Scenarios: Covers five daily environments: Bedroom, Dining Room, Living Room, Fitness Room, and Office.
Compared to previous datasets like GTA-IM (synthetic) or PROX (static scenes), HOI-M3 is the first to offer real-world, dynamic, multi-person, and multi-object tracking with high-resolution ground truth.
Key Results
The authors benchmarked their monocular capture method against state-of-the-art (SOTA) single-HOI models like PHOSA and CHORE.
- Accuracy: In multi-person scenarios, the HOI-M3 baseline achieved a PCK (Percentage of Correct Keypoints) of 68.5%, vastly outperforming CHORE (10.4%) and PHOSA (43.9%), which struggle when the scene becomes crowded.
- Object Tracking: Their method reduced the "Vertex-to-Vertex" (V2V) error for objects to 297.8mm, compared to over 690mm for PHOSA.
- Generation: Their diffusion-based model successfully synthesized semantically meaningful motions (e.g., two people sitting at a table together) that respected the physical boundaries of the objects.
Limitations and Future Work
While a major step forward, the authors note several constraints. The capture is currently limited to indoor lab settings with controlled illumination. Expanding this to "in-the-wild" outdoor environments remains a significant challenge due to the hardware requirements of 42 cameras and IMU-embedded objects. Future work will likely focus on moving these capabilities to unstructured environments and improving the "common sense" physics of generated interactions.
Summary updated