Abstract
PhysCap is the first real-time (25 fps) system that captures physically plausible 3D human motion from a single RGB camera. By combining deep learning with a physics-based dynamics model, it eliminates common artifacts like foot sliding and floor penetration, producing stable animations suitable for immediate use in VR and gaming.
Introduction
Capturing 3D human motion from a single 2D video (monocular mocap) is a "severely ill-posed" problem due to the inherent depth ambiguity of flat images. While modern AI models have become proficient at estimating joint positions, they are often "physics-blind." Purely kinematic methods—those based solely on geometry—frequently produce results that look unnatural to the human eye. Common artifacts include "foot skating" (feet sliding across the floor), floor penetration, unnatural leaning, and jittery movements caused by frame-by-frame estimation errors.
PhysCap addresses these issues by integrating physical laws into the capture process, ensuring that the resulting motion is not only geometrically accurate but also physically plausible.
Methodology
The authors propose a three-stage pipeline that bridges the gap between computer vision and physical simulation, allowing for real-time processing on consumer-grade hardware.
- Kinematic Pose Estimation: A Convolutional Neural Network (CNN) estimates 2D and 3D joint positions. An Inverse Kinematics (IK) step then fits these to a skeletal model to generate initial joint angles, providing a "rough draft" of the motion.
- Foot Contact & Motion State Detection: A specialized temporal CNN analyzes 2D keypoints to determine if a foot (heel or toe) is touching the ground and whether the subject is stationary or moving.
- Physics-Based Global Pose Optimization: This core innovation treats the human as a "floating-base" physical character. Using Proportional-Derivative (PD) controllers, the system calculates the torques required for the physical character to follow the kinematic "rough draft" while enforcing constraints for gravity, collisions, and Ground Reaction Forces (GRF).
By solving for these forces in a two-step optimization process, the system achieves a performance of 25 fps on a standard PC (e.g., RTX 2070), making it viable for live applications.
Experimental Setup
PhysCap was evaluated on several standard benchmarks, including Human3.6M, MPI-INF-3DHP, and DeepCap. To demonstrate "in-the-wild" performance, the authors recorded six new outdoor sequences in diverse environments like forests and backyards where ground truth data is typically unavailable. These tests confirmed the system's robustness to varying lighting and backgrounds.
Results and Discussion
PhysCap significantly outperforms state-of-the-art kinematic methods in physical metrics. It reduced Mean Penetration Error (MPE) by approximately 30% and improved the Percentage of Non-Penetration (PNP) from ~45% to nearly 93%. On standard benchmarks like Human3.6M, it achieved competitive accuracy while providing much smoother and more stable results. In a user study, participants preferred PhysCap’s reconstructions in 89.2% of cases, citing the lack of jitter and more natural-looking balance.
A key insight of the work is using physics as a regularizer. Instead of simple smoothing, physics acts as a filter that rejects impossible poses. Furthermore, the authors achieved real-time speeds by decoupling the estimation of Ground Reaction Forces from the pose optimization, allowing for rapid computation without significant loss in accuracy.
Conclusion and Future Work
While PhysCap represents a major step forward for monocular motion capture, it currently assumes a calibrated, flat floor plane and only models foot-to-floor contact. It can struggle with self-occlusion or complex environment interactions like sitting or leaning against walls. Future work aims to extend the model to handle uneven terrain, self-collisions, and more diverse object interactions.
Summary updated