Abstract
Estimating full-body 3D human pose from a single head-mounted display (HMD) is a critical challenge for immersive virtual and augmented reality. We present xR-EgoPose, a robust framework that utilizes a downward-looking fisheye camera to predict 3D poses. By employing a novel dual-branch auto-encoder that explicitly models joint uncertainty through 2D heatmaps and training on a massive new photorealistic synthetic dataset, our approach achieves state-of-the-art accuracy under extreme perspective distortion and self-occlusion.
Introduction
In Virtual and Augmented Reality (xR), the ability for users to perceive their own body is fundamental to immersion and social presence. Current HMDs typically track only the head and hands, relying on inverse kinematics to estimate the rest of the body, which often results in unnatural movements.
Egocentric 3D pose estimation from a camera mounted on the HMD (approximately 2cm from the user's nose) presents significant hurdles:
- Extreme Distortion: Fisheye lenses create a vast resolution disparity between the upper and lower body.
- Severe Self-Occlusion: The torso frequently obstructs the view of the legs and feet from this high-angle perspective.
- Data Scarcity: There is a lack of large-scale datasets with accurate 3D ground truth from this specific viewpoint.
Methods
The xR-EgoPose framework utilizes a two-stage "2D-to-3D" pipeline that separates image feature extraction from geometric lifting.
2D Heatmap Estimation
A ResNet-101 architecture processes the RGB fisheye image to predict 2D heatmaps for 15 body joints. These heatmaps are crucial as they represent the spatial probability distribution of each joint.
Dual-Branch Auto-encoder
The core innovation is a dual-branch auto-encoder that takes 2D heatmaps as input. The encoder produces a latent representation ($z$), which is then processed by two distinct decoder branches:
- 3D Pose Branch: Regresses the 3D joint coordinates.
- 2D Reconstruction Branch: Reconstructs the original 2D heatmaps from the latent vector.
By forcing the network to reconstruct the 2D heatmaps, the latent space is compelled to encode joint uncertainty. For instance, if a foot is occluded, the resulting "blurry" heatmap informs the model's 3D estimation, allowing it to leverage probabilistic information rather than just point coordinates.
Datasets
The study introduces two key datasets:
- xR-EgoPose (Synthetic): A large-scale dataset of 383,000 frames featuring diverse characters, actions, and lighting conditions.
- xR-EgoPose$^R$ (Real): A real-world evaluation set (~10k frames) captured with an HMD-mounted camera and motion capture ground truth.
Results
The proposed method demonstrates significant improvements over existing benchmarks:
- Accuracy: On the Mo2Cap2 dataset, xR-EgoPose outperformed state-of-the-art methods by over 21.6% in indoor settings and 25.4% in outdoor settings.
- Ablation Study: The dual-branch architecture proved essential; including the reconstruction branch reduced average error from 130.4mm to 58.2mm.
- Real-World Transfer: The model achieved an average error of 61.71mm on real-world data, validating the effectiveness of training on photorealistic synthetic data.
Conclusion
xR-EgoPose provides a robust solution for egocentric pose estimation by treating uncertainty as a feature. While the system handles significant occlusion, future work will explore multi-camera configurations to eliminate blind spots and incorporate temporal modeling to improve prediction smoothness.
Summary updated