xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera

Denis Tomè, Patrick Peluse, Lourdes Agapito, Hernán Badino

Abstract

Estimating full-body 3D human pose from a single head-mounted display (HMD) is a critical challenge for immersive virtual and augmented reality. We present xR-EgoPose, a robust framework that utilizes a downward-looking fisheye camera to predict 3D poses. By employing a novel dual-branch auto-encoder that explicitly models joint uncertainty through 2D heatmaps and training on a massive new photorealistic synthetic dataset, our approach achieves state-of-the-art accuracy under extreme perspective distortion and self-occlusion.

Introduction

In Virtual and Augmented Reality (xR), the ability for users to perceive their own body is fundamental to immersion and social presence. Current HMDs typically track only the head and hands, relying on inverse kinematics to estimate the rest of the body, which often results in unnatural movements.

Overview of the xR-EgoPose pipeline: from HMD-mounted fisheye capture (a-c) to 3D pose reconstruction (d-f).

Egocentric 3D pose estimation from a camera mounted on the HMD (approximately 2cm from the user's nose) presents significant hurdles:

  1. Extreme Distortion: Fisheye lenses create a vast resolution disparity between the upper and lower body.
  2. Severe Self-Occlusion: The torso frequently obstructs the view of the legs and feet from this high-angle perspective.
  3. Data Scarcity: There is a lack of large-scale datasets with accurate 3D ground truth from this specific viewpoint.

Methods

The xR-EgoPose framework utilizes a two-stage "2D-to-3D" pipeline that separates image feature extraction from geometric lifting.

2D Heatmap Estimation

A ResNet-101 architecture processes the RGB fisheye image to predict 2D heatmaps for 15 body joints. These heatmaps are crucial as they represent the spatial probability distribution of each joint.

Dual-Branch Auto-encoder

The core innovation is a dual-branch auto-encoder that takes 2D heatmaps as input. The encoder produces a latent representation ($z$), which is then processed by two distinct decoder branches:

  • 3D Pose Branch: Regresses the 3D joint coordinates.
  • 2D Reconstruction Branch: Reconstructs the original 2D heatmaps from the latent vector.
Predicted 2D joint heatmaps (e) capturing spatial uncertainty, which the auto-encoder uses to resolve ambiguities in 3D pose estimation.

By forcing the network to reconstruct the 2D heatmaps, the latent space is compelled to encode joint uncertainty. For instance, if a foot is occluded, the resulting "blurry" heatmap informs the model's 3D estimation, allowing it to leverage probabilistic information rather than just point coordinates.

Datasets

The study introduces two key datasets:

  • xR-EgoPose (Synthetic): A large-scale dataset of 383,000 frames featuring diverse characters, actions, and lighting conditions.
  • xR-EgoPose$^R$ (Real): A real-world evaluation set (~10k frames) captured with an HMD-mounted camera and motion capture ground truth.

Results

The proposed method demonstrates significant improvements over existing benchmarks:

  • Accuracy: On the Mo2Cap2 dataset, xR-EgoPose outperformed state-of-the-art methods by over 21.6% in indoor settings and 25.4% in outdoor settings.
  • Ablation Study: The dual-branch architecture proved essential; including the reconstruction branch reduced average error from 130.4mm to 58.2mm.
  • Real-World Transfer: The model achieved an average error of 61.71mm on real-world data, validating the effectiveness of training on photorealistic synthetic data.

Conclusion

xR-EgoPose provides a robust solution for egocentric pose estimation by treating uncertainty as a feature. While the system handles significant occlusion, future work will explore multi-camera configurations to eliminate blind spots and incorporate temporal modeling to improve prediction smoothness.

Summary updated

Abstract

We present a new solution to egocentric 3D body pose estimation from monocular images captured from a downward looking fish-eye camera installed on the rim of a head mounted virtual reality device. This unusual viewpoint, just 2 cm. away from the user's face, leads to images with unique visual appearance, characterized by severe self-occlusions and strong perspective distortions that result in a drastic difference in resolution between lower and upper body. Our contribution is two-fold. Firstly, we propose a new encoder-decoder architecture with a novel dual branch decoder designed specifically to account for the varying uncertainty in the 2D joint locations. Our quantitative evaluation, both on synthetic and real-world datasets, shows that our strategy leads to substantial improvements in accuracy over state of the art egocentric pose estimation approaches. Our second contribution is a new large-scale photorealistic synthetic dataset - xR-EgoPose - offering 383K frames of high quality renderings of people with a diversity of skin tones, body shapes, clothing, in a variety of backgrounds and lighting conditions, performing a range of actions. Our experiments show that the high variability in our new synthetic training corpus leads to good generalization to real world footage and to state of the art results on real world datasets with ground truth. Moreover, an evaluation on the Human3.6M benchmark shows that the performance of our method is on par with top performing approaches on the more classic problem of 3D human pose from a third person viewpoint.
BibTeX
@inproceedings{tome2019xr,
  author = {Denis Tomè and Patrick Peluse and Lourdes Agapito and Hernán Badino},
  title = {{xR-EgoPose: Egocentric 3D Human Pose from an HMD Camera}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=ixKpzOVdwc},
}
Figure