Abstract
The estimation of 3D human motion from video has progressed rapidly but current methods still have several key limitations. First, most methods estimate the human in camera coordinates. Second, prior work on estimating humans in global coordinates often assumes a flat ground plane and produces foot sliding. Third, the most accurate methods rely on computationally expensive optimization pipelines, limiting their use to offline applications. Finally, existing video-based methods are surprisingly less accurate than single-frame methods. We address these limitations with WHAM (World-grounded Humans with Accurate Motion), which accurately and efficiently reconstructs 3D human motion in a global coordinate system from video. WHAM learns to lift 2D keypoint sequences to 3D using motion capture data and fuses this with video features, integrating motion context and visual information. WHAM exploits camera angular velocity estimated from a SLAM method together with human motion to estimate the body's global trajectory. We combine this with a contact-aware trajectory refinement method that lets WHAM capture human motion in diverse conditions, such as climbing stairs. WHAM outperforms all existing 3D human motion recovery methods across multiple in-the-wild benchmarks.
Introduction
Estimating 3D human pose and shape (HPS) from video is a fundamental challenge in computer vision. Despite recent progress, existing methods often struggle to reconstruct motion accurately in "world coordinates" when the camera is moving. Common issues include camera-centricity, where the human appears to drift relative to the environment; a trade-off between speed and accuracy, where the best results require slow offline optimization; and temporal inconsistency, where video-based models fail to outperform single-frame counterparts.
Methodology
WHAM (World-grounded Humans with Accurate Motion) is an online, recursive framework designed for real-time, accurate 3D reconstruction. Its architecture consists of several key components:
- Feature Integration: WHAM utilizes a Feature Integrator that combines temporal motion context from 2D keypoint sequences with dense visual features from a pretrained image encoder. This fusion helps disambiguate complex poses that sparse keypoints alone might miss.
- Global Trajectory Decoder: To decouple human motion from camera movement, WHAM integrates camera angular velocity (sourced from SLAM or a device's gyroscope) into the motion context. This enables accurate prediction of global orientation and velocity.
- Contact-Aware Trajectory Refinement: To eliminate "foot sliding," the model predicts foot-ground contact probabilities. A refinement network then adjusts the trajectory to ensure that feet remain stationary when in contact with the ground, even on non-planar surfaces like stairs.
- Two-Stage Training: The model is pretrained on the AMASS motion capture dataset using synthetic projections and then fine-tuned on real-world video datasets to handle visual complexities.
Evaluation and Results
WHAM was evaluated on several benchmarks, including 3DPW, RICH, and EMDB. It demonstrates state-of-the-art performance in both camera and world coordinates.
Key results include:
- State-of-the-Art Accuracy: WHAM outperforms leading methods like TRACE and SLAHMR across all major benchmarks (3DPW, RICH, EMDB).
- Blazing Fast Performance: The core network runs at 200 FPS on an A100 GPU. The full pipeline, including detection and keypoint estimation, achieves ~9 FPS for online inference and ~50 FPS in batch mode.
- Reduced Drift: On the EMDB dataset, WHAM significantly reduces Root Translation Error (RTE) and foot sliding compared to regression-based competitors.
Discussion and Conclusion
WHAM demonstrates that integrating motion context with visual evidence and camera dynamics can yield high-fidelity, world-grounded 3D reconstructions in real-time. By leveraging hardware sensors like gyroscopes and implementing contact-aware refinement, it overcomes the limitations of previous optimization-heavy or camera-centric approaches.
While highly effective, the authors note that WHAM faces challenges with out-of-distribution actions (e.g., cycling) and complex occlusions. Future work will focus on deeper integration with SLAM systems to enforce scene consistency and developing fully optimized mobile implementations.
Summary updated