Abstract
Understanding human motion and social interactions from an egocentric perspective is critical for the development of immersive Augmented Reality (AR) and social robotics. However, existing datasets often lack the high-quality 3D ground truth necessary to train robust models for these "first-person" views. We introduce EgoBody, a large-scale dataset specifically designed to capture the 3D shape and motion of social interaction partners from a head-mounted device (HMD). EgoBody provides high-quality 3D ground truth for 125 sequences across diverse social scenarios, synchronized with multi-modal data including eye gaze and depth. Our analysis reveals a substantial performance gap—the "Ego Gap"—when applying state-of-the-art third-person models to egocentric data. We demonstrate that fine-tuning on EgoBody significantly reduces error and improves robustness, establishing a new benchmark for egocentric human sensing.
Introduction
Most current 3D human pose and shape estimation (3DHPS) models are trained on "third-person" datasets, where subjects are typically fully visible from a distance. However, applications in AR/VR and assistive robotics require machines to interpret the world from an egocentric view. This perspective introduces unique challenges: severe body truncation due to proximity, significant motion blur from head movements, and a general lack of annotated 3D data for complex two-person interactions.
The EgoBody Dataset
To address these challenges, we developed a sophisticated capture system combining a Microsoft HoloLens 2 (worn by the camera wearer) with a multi-Kinect rig (providing a third-person "god's eye" view). This setup allowed us to record 220,000 frames across 125 sequences involving 36 subjects.
Methodology
Our pipeline ensures high-fidelity data through several key innovations:
- Keypoint-Based Refinement: We align HMD data with the Kinect rig to correct for hardware calibration drift.
- SMPL-X Fitting: We utilize a marker-less motion capture pipeline to fit expressive 3D body meshes (SMPL-X) to multi-view RGB-D data, providing accurate ground truth for both participants.
- Multi-Modal Integration: Beyond video, we capture eye gaze, head tracking, hand tracking, and depth data, all situated within reconstructed 3D scene contexts of 15 indoor environments.
Key Insights: The "Ego Gap"
Our research identifies a significant performance disparity, termed the "Ego Gap." State-of-the-art models perform significantly worse on egocentric data, with 3D joint errors averaging 77% higher than on traditional third-person benchmarks like 3DPW.
Gaze-Driven Truncation
Analysis of eye gaze data reveals that humans focus heavily on an interaction partner's face and upper body. This behavioral pattern explains the frequent truncation of lower bodies in egocentric video; we simply do not look at a partner's feet during social exchange. This natural bias in the field of view makes domain adaptation essential for models intended for HMD applications.
Experimental Results
We benchmarked several top-tier models, including SPIN, METRO, and EFT. While all struggled initially with the EgoBody test set, fine-tuning on our dataset yielded dramatic improvements:
- Error Reduction: Mean Per-Joint Position Error (MPJPE) decreased by up to 42%.
- Increased Robustness: Fine-tuned models showed significantly better handling of motion blur and partial body invisibility.
- Generalization: Models trained on EgoBody demonstrated improved performance on independent egocentric datasets like You2Me, confirming the dataset's broad utility for domain adaptation.
Conclusion and Future Work
EgoBody bridges the gap between third-person computer vision and the requirements of egocentric sensing. While current limitations include minor temporal misalignment due to hardware synchronization constraints, the dataset provides a robust foundation for future research. Potential directions include expanding to multi-participant interactions, integrating audio, and utilizing natural language descriptions to support multi-modal AI development in social contexts.
Summary updated