EgoBody: Human Body Shape, Motion and Social Interactions from Head-Mounted Devices

Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Marc Pollefeys, Federica Bogo, Siyu Tang

Abstract

Understanding human motion and social interactions from an egocentric perspective is critical for the development of immersive Augmented Reality (AR) and social robotics. However, existing datasets often lack the high-quality 3D ground truth necessary to train robust models for these "first-person" views. We introduce EgoBody, a large-scale dataset specifically designed to capture the 3D shape and motion of social interaction partners from a head-mounted device (HMD). EgoBody provides high-quality 3D ground truth for 125 sequences across diverse social scenarios, synchronized with multi-modal data including eye gaze and depth. Our analysis reveals a substantial performance gap—the "Ego Gap"—when applying state-of-the-art third-person models to egocentric data. We demonstrate that fine-tuning on EgoBody significantly reduces error and improves robustness, establishing a new benchmark for egocentric human sensing.

Introduction

Most current 3D human pose and shape estimation (3DHPS) models are trained on "third-person" datasets, where subjects are typically fully visible from a distance. However, applications in AR/VR and assistive robotics require machines to interpret the world from an egocentric view. This perspective introduces unique challenges: severe body truncation due to proximity, significant motion blur from head movements, and a general lack of annotated 3D data for complex two-person interactions.

The EgoBody Dataset

To address these challenges, we developed a sophisticated capture system combining a Microsoft HoloLens 2 (worn by the camera wearer) with a multi-Kinect rig (providing a third-person "god's eye" view). This setup allowed us to record 220,000 frames across 125 sequences involving 36 subjects.

Methodology

Our pipeline ensures high-fidelity data through several key innovations:

  1. Keypoint-Based Refinement: We align HMD data with the Kinect rig to correct for hardware calibration drift.
  2. SMPL-X Fitting: We utilize a marker-less motion capture pipeline to fit expressive 3D body meshes (SMPL-X) to multi-view RGB-D data, providing accurate ground truth for both participants.
  3. Multi-Modal Integration: Beyond video, we capture eye gaze, head tracking, hand tracking, and depth data, all situated within reconstructed 3D scene contexts of 15 indoor environments.
The EgoBody dataset provides comprehensive 3D ground truth, aligning egocentric views with 3D scene context and body meshes across diverse social interactions.

Key Insights: The "Ego Gap"

Our research identifies a significant performance disparity, termed the "Ego Gap." State-of-the-art models perform significantly worse on egocentric data, with 3D joint errors averaging 77% higher than on traditional third-person benchmarks like 3DPW.

Gaze-Driven Truncation

Analysis of eye gaze data reveals that humans focus heavily on an interaction partner's face and upper body. This behavioral pattern explains the frequent truncation of lower bodies in egocentric video; we simply do not look at a partner's feet during social exchange. This natural bias in the field of view makes domain adaptation essential for models intended for HMD applications.

EgoBody captures detailed interaction data, including hand-object contact and gaze-driven focus points (red heatmaps), which are essential for understanding social dynamics.

Experimental Results

We benchmarked several top-tier models, including SPIN, METRO, and EFT. While all struggled initially with the EgoBody test set, fine-tuning on our dataset yielded dramatic improvements:

  • Error Reduction: Mean Per-Joint Position Error (MPJPE) decreased by up to 42%.
  • Increased Robustness: Fine-tuned models showed significantly better handling of motion blur and partial body invisibility.
  • Generalization: Models trained on EgoBody demonstrated improved performance on independent egocentric datasets like You2Me, confirming the dataset's broad utility for domain adaptation.

Conclusion and Future Work

EgoBody bridges the gap between third-person computer vision and the requirements of egocentric sensing. While current limitations include minor temporal misalignment due to hardware synchronization constraints, the dataset provides a robust foundation for future research. Potential directions include expanding to multi-participant interactions, integrating audio, and utilizing natural language descriptions to support multi-modal AI development in social contexts.

Summary updated

Abstract

Understanding social interactions from egocentric views is crucial for many applications, ranging from assistive robotics to AR/VR. Key to reasoning about interactions is to understand the body pose and motion of the interaction partner from the egocentric view. However, research in this area is severely hindered by the lack of datasets. Existing datasets are limited in terms of either size, capture/annotation modalities, ground-truth quality, or interaction diversity. We fill this gap by proposing EgoBody, a novel large-scale dataset for human pose, shape and motion estimation from egocentric views, during interactions in complex 3D scenes. We employ Microsoft HoloLens2 headsets to record rich egocentric data streams (including RGB, depth, eye gaze, head and hand tracking). To obtain accurate 3D ground truth, we calibrate the headset with a multi-Kinect rig and fit expressive SMPL-X body meshes to multi-view RGB-D frames, reconstructing 3D human shapes and poses relative to the scene, over time. We collect 125 sequences, spanning diverse interaction scenarios, and propose the first benchmark for 3D full-body pose and shape estimation of the social partner from egocentric views. We extensively evaluate state-of-the-art methods, highlight their limitations in the egocentric scenario, and address such limitations leveraging our high-quality annotations. Data and code are available at https://sanweiliti.github.io/egobody/egobody.html.
BibTeX
@inproceedings{zhang2022egobody,
  author = {Siwei Zhang and Qianli Ma and Yan Zhang and Zhiyin Qian and Marc Pollefeys and Federica Bogo and Siyu Tang},
  title = {{EgoBody: Human Body Shape, Motion and Social Interactions from Head-Mounted Devices}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-20068-7\_11},
}
Figure