TL;DR
This paper introduces EgoBody, the first large-scale dataset for estimating the 3D pose and shape of people interacting in egocentric views captured by head-mounted devices. They also establish a benchmark for this task, demonstrating that existing methods struggle significantly with egocentric data and showing how their dataset can be used to improve their performance.
What Problem Does This Paper Address?
Understanding human social interactions is crucial for applications like assistive robotics and AR/VR. A key component of this is accurately estimating the 3D body pose and motion of the person you're interacting with, as seen from your own perspective (egocentric view), typically from a head-mounted device (HMD). However, this area has been severely limited by a lack of suitable datasets. Existing datasets are often too small, lack diverse interaction scenarios, have low-quality annotations, or don't capture the unique challenges of egocentric views, such as severe body truncations, motion blur, and people entering/leaving the frame. This paper aims to fill this gap by creating a comprehensive dataset and establishing a benchmark for egocentric 3D human pose and shape estimation during interactions.
Methods
The authors developed a novel system for capturing and processing egocentric interaction data. They used Microsoft HoloLens2 headsets to record rich egocentric streams, including RGB, depth, eye gaze, head, and hand tracking. To obtain accurate 3D ground truth, they employed a multi-Kinect rig for third-person view capture. A key part of their method involves a sophisticated calibration process between the HoloLens2 and Kinects, which is refined using body keypoints. They then fit expressive SMPL-X body meshes to multi-view RGB-D frames to reconstruct 3D human shapes and poses relative to the scene over time. This process allows them to generate high-quality, per-frame 3D ground truth for both interacting subjects.
Datasets
The paper introduces EgoBody, a new large-scale dataset specifically designed for egocentric 3D human pose and shape estimation during interactions. It comprises 125 sequences from 36 subjects across 15 indoor scenes, capturing diverse social interaction scenarios. EgoBody includes:
- Multi-view (MV)Set: Synchronized RGB and depth frames from multiple Azure Kinect cameras, providing third-person views.
- EgoSet: Egocentric RGB, depth, head, hand, and eye gaze tracking data from a Microsoft HoloLens2 headset.
- EgoSet-interactee: A subset of EgoSet frames where the interaction partner is visible.
EgoBody also serves as the first benchmark for 3D full-body pose and shape estimation of an interactee from egocentric views.
Key Insights and Ideas
- Egocentric data is fundamentally different: State-of-the-art 3D human pose and shape estimation (3DHPS) methods, trained on third-person views, perform significantly worse on egocentric data due to challenges like severe body truncation and motion blur.
- Eye gaze is a rich signal for interaction: The inclusion of eye gaze tracking in EgoBody opens up new avenues for studying human attention during interactions, which is shown to be highly focused on the interactee.
- Fine-tuning on egocentric data is crucial: Simply training existing methods on third-person data is insufficient. Fine-tuning these models on the EgoBody dataset leads to substantial improvements in accuracy and robustness for egocentric scenarios.
- Body part visibility matters: Lower body parts are more frequently truncated in egocentric views, directly impacting estimation accuracy. This highlights the need for methods robust to partial visibility.
Key Results
- The EgoBody dataset contains 219,731 synchronized frames.
- Existing state-of-the-art 3DHPS methods show a significant performance drop (e.g., 77% higher 3D joint error) on EgoBody compared to third-person datasets like 3DPW.
- Fine-tuning three recent methods (SPIN, METRO, EFT) on EgoBody resulted in substantial error reductions: SPIN-ft achieved a 42% lower MPJPE, METRO-ft a 36% lower MPJPE, and EFT-ft a 18% lower MPJPE compared to their original versions.
- The fine-tuned models also demonstrated improved robustness to motion blur and partial visibility.
- The EgoBody training set also improved performance on the You2Me egocentric dataset, indicating generalizability.
Limitations and Future Work
- Temporal Misalignment: While efforts were made to synchronize devices, small temporal misalignments can occur, especially during fast motions, due to the lack of hardware synchronization solutions between HoloLens2 and Kinects. This is an inherent challenge for egocentric systems.
- Data Augmentation Limits: Standard data augmentation techniques like aggressive cropping or motion blur augmentation are not sufficient to fully bridge the performance gap for egocentric scenarios.
- Future Work: The authors suggest enriching the dataset with more participants, richer data modalities (e.g., audio), and more detailed motion annotations (e.g., natural language descriptions) to further advance the understanding of egocentric human motions, behaviors, and social interactions.
Summary updated