Abstract
We present the first real-time human performance capture approach that reconstructs dense, space-time coherent deforming geometry of entire humans in general everyday clothing from just a single RGB video. We propose a novel two-stage analysis-by-synthesis optimization whose formulation and implementation are designed for high performance. In the first stage, a skinned template model is jointly fitted to background subtracted input video, 2D and 3D skeleton joint positions found using a deep neural network, and a set of sparse facial landmark detections. In the second stage, dense non-rigid 3D deformations of skin and even loose apparel are captured based on a novel real-time capable algorithm for non-rigid tracking using dense photometric and silhouette constraints. Our novel energy formulation leverages automatically identified material regions on the template to model the differing non-rigid deformation behavior of skin and apparel. The two resulting non-linear optimization problems per-frame are solved with specially-tailored data-parallel Gauss-Newton solvers. In order to achieve real-time performance of over 25Hz, we design a pipelined parallel architecture using the CPU and two commodity GPUs. Our method is the first real-time monocular approach for full-body performance capture, yielding comparable accuracy with off-line techniques while being orders of magnitude faster.
Introduction
Capturing the 3D motion and shape of a human in motion is a cornerstone of VR/AR, gaming, and telepresence. Traditionally, high-quality "performance capture"—which includes the deformation of clothing, not just bone movement—required expensive multi-camera studios or depth sensors. While recent deep learning methods can estimate 3D skeletons from a single image, they often ignore clothing and produce low-fidelity skeletal models. Conversely, methods that capture dense surface details are typically "offline," taking minutes to process a single second of video.
ReTiCaM (also known as LiveCap) addresses this "speed vs. fidelity" trade-off, enabling high-quality, dense 3D reconstruction from a single color stream at interactive frame rates (25+ FPS).
Methodology
The system utilizes a two-stage analysis-by-synthesis optimization pipeline to bridge the gap between skeletal tracking and dense surface refinement.
Stage I: Skeletal Pose Estimation
The system begins with a pre-scanned "template" of the user. A Convolutional Neural Network (CNN) detects 2D and 3D joint positions and facial landmarks. A skeletal rig is then fitted to these detections and the user's silhouette to establish the global pose.
Stage II: Non-Rigid Surface Refinement
To capture fine details like the folds of a shirt or the swing of a skirt, the system performs dense non-rigid deformation. This stage uses a novel energy formulation considering photometric alignment (matching model colors to video pixels) and silhouette constraints. A key innovation is material-aware regularization: the system automatically segments the template into different material classes (e.g., skin vs. loose clothing) and applies different "stiffness" weights to each, allowing cloth to deform more freely than the head or limbs.
Hardware Acceleration
To achieve real-time speeds, the authors designed a pipelined parallel architecture. By distributing tasks across the CPU and two commodity GPUs—running the CNN on one and the non-rigid solver on another in a "rolling" manner—the system ensures the optimization loop finishes in under 40ms.
Results and Evaluation
The method was evaluated on a new benchmark dataset featuring over 20 minutes of video across 11 scenarios with 7 different subjects wearing various apparel. Quantitatively, the system achieves an Intersection over Union (IoU) for silhouettes of approximately 88%, which is only ~4% lower than state-of-the-art offline methods that are orders of magnitude slower. Qualitatively, the system successfully tracks challenging motions like 360-degree spins and jumping jacks, maintaining a dense mesh that accurately overlays the input video.
Conclusion and Future Work
ReTiCaM represents a significant step forward in monocular performance capture, offering a real-time solution for dense 3D reconstruction. Current limitations include sensitivity to extreme poses not present in training data, an inability to handle mid-stream topological changes (like removing a jacket), and challenges with self-occlusion. Future research will focus on integrating hand gesture tracking and more sophisticated lighting estimation to further enhance realism.
Summary updated