PhysCap: physically plausible monocular 3D motion capture in real time

Soshi Shimada, Vladislav Golyanik, Weipeng Xu, Christian Theobalt

Abstract

PhysCap is the first real-time (25 fps) system that captures physically plausible 3D human motion from a single RGB camera. By combining deep learning with a physics-based dynamics model, it eliminates common artifacts like foot sliding and floor penetration, producing stable animations suitable for immediate use in VR and gaming.

Introduction

Capturing 3D human motion from a single 2D video (monocular mocap) is a "severely ill-posed" problem due to the inherent depth ambiguity of flat images. While modern AI models have become proficient at estimating joint positions, they are often "physics-blind." Purely kinematic methods—those based solely on geometry—frequently produce results that look unnatural to the human eye. Common artifacts include "foot skating" (feet sliding across the floor), floor penetration, unnatural leaning, and jittery movements caused by frame-by-frame estimation errors.

PhysCap captures 3D motion from a single RGB camera in real-time, correcting common kinematic artifacts.

PhysCap addresses these issues by integrating physical laws into the capture process, ensuring that the resulting motion is not only geometrically accurate but also physically plausible.

Methodology

The authors propose a three-stage pipeline that bridges the gap between computer vision and physical simulation, allowing for real-time processing on consumer-grade hardware.

The two-stage pipeline for kinematic pose estimation and foot contact prediction.
  1. Kinematic Pose Estimation: A Convolutional Neural Network (CNN) estimates 2D and 3D joint positions. An Inverse Kinematics (IK) step then fits these to a skeletal model to generate initial joint angles, providing a "rough draft" of the motion.
  2. Foot Contact & Motion State Detection: A specialized temporal CNN analyzes 2D keypoints to determine if a foot (heel or toe) is touching the ground and whether the subject is stationary or moving.
  3. Physics-Based Global Pose Optimization: This core innovation treats the human as a "floating-base" physical character. Using Proportional-Derivative (PD) controllers, the system calculates the torques required for the physical character to follow the kinematic "rough draft" while enforcing constraints for gravity, collisions, and Ground Reaction Forces (GRF).

By solving for these forces in a two-step optimization process, the system achieves a performance of 25 fps on a standard PC (e.g., RTX 2070), making it viable for live applications.

Experimental Setup

PhysCap was evaluated on several standard benchmarks, including Human3.6M, MPI-INF-3DHP, and DeepCap. To demonstrate "in-the-wild" performance, the authors recorded six new outdoor sequences in diverse environments like forests and backyards where ground truth data is typically unavailable. These tests confirmed the system's robustness to varying lighting and backgrounds.

Results and Discussion

PhysCap significantly outperforms state-of-the-art kinematic methods in physical metrics. It reduced Mean Penetration Error (MPE) by approximately 30% and improved the Percentage of Non-Penetration (PNP) from ~45% to nearly 93%. On standard benchmarks like Human3.6M, it achieved competitive accuracy while providing much smoother and more stable results. In a user study, participants preferred PhysCap’s reconstructions in 89.2% of cases, citing the lack of jitter and more natural-looking balance.

A key insight of the work is using physics as a regularizer. Instead of simple smoothing, physics acts as a filter that rejects impossible poses. Furthermore, the authors achieved real-time speeds by decoupling the estimation of Ground Reaction Forces from the pose optimization, allowing for rapid computation without significant loss in accuracy.

Conclusion and Future Work

While PhysCap represents a major step forward for monocular motion capture, it currently assumes a calibrated, flat floor plane and only models foot-to-floor contact. It can struggle with self-occlusion or complex environment interactions like sitting or leaning against walls. Future work aims to extend the model to handle uneven terrain, self-collisions, and more diverse object interactions.

Summary updated

Abstract

Marker-less 3D human motion capture from a single colour camera has seen significant progress. However, it is a very challenging and severely ill-posed problem. In consequence, even the most accurate state-of-the-art approaches have significant limitations. Purely kinematic formulations on the basis of individual joints or skeletons, and the frequent frame-wise reconstruction in state-of-the-art methods greatly limit 3D accuracy and temporal stability compared to multi-view or marker-based motion capture. Further, captured 3D poses are often physically incorrect and biomechanically implausible, or exhibit implausible environment interactions (floor penetration, foot skating, unnatural body leaning and strong shifting in depth), which is problematic for any use case in computer graphics. We, therefore, present PhysCap, the first algorithm for physically plausible, real-time and marker-less human 3D motion capture with a single colour camera at 25 fps. Our algorithm first captures 3D human poses purely kinematically. To this end, a CNN infers 2D and 3D joint positions, and subsequently, an inverse kinematics step finds space-time coherent joint angles and global 3D pose. Next, these kinematic reconstructions are used as constraints in a real-time physics-based pose optimiser that accounts for environment constraints (e.g., collision handling and floor placement), gravity, and biophysical plausibility of human postures. Our approach employs a combination of ground reaction force and residual force for plausible root control, and uses a trained neural network to detect foot contact events in images. Our method captures physically plausible and temporally stable global 3D human motion, without physically implausible postures, floor penetrations or foot skating, from video in real time and in general scenes. PhysCap achieves state-of-the-art accuracy on established pose benchmarks, and we propose new metrics to demonstrate the improved physical plausibility and temporal stability.
BibTeX
@article{shimada2020physcap,
  author = {Soshi Shimada and Vladislav Golyanik and Weipeng Xu and Christian Theobalt},
  title = {{PhysCap: physically plausible monocular 3D motion capture in real time}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  doi = {10.1145/3414685.3417877},
}
Figure