TL;DR
This paper introduces LiveCap, the first real-time system for capturing dense, 3D human performance from a single monocular video. It achieves this by combining a novel two-stage optimization approach with a pipelined GPU architecture, enabling high-fidelity reconstruction of both body shape and loose clothing at over 25 FPS.
What Problem Does This Paper Address?
Creating realistic 3D virtual human characters for applications like games, VR/AR, and visual effects is currently a labor-intensive process, often requiring months of work by artists. While multi-camera performance capture methods can produce high-quality results, they are expensive and require controlled environments. Existing markerless methods that use single cameras are typically limited to capturing coarse skeletal motion, failing to reconstruct the detailed shape and deformation of clothing. The challenge is to achieve high-quality, dense, and temporally coherent 3D human performance capture in real-time using only a single, readily available RGB camera, which is a significant underconstrained problem.
Methods
LiveCap employs a two-stage "analysis-by-synthesis" optimization pipeline designed for real-time performance.
- Stage I: Skeletal Pose Estimation: This stage estimates the articulated 3D pose of the underlying kinematic skeleton. It fits a rigged template model to:
- 2D and 3D joint positions predicted by a deep neural network (VNect).
- Sparse facial landmark detections (to improve head orientation).
- The foreground silhouette of the person.
- Temporal and anatomical regularization constraints are used to ensure smooth and plausible motion.
- Stage II: Non-Rigid Surface Registration: This stage captures dense, non-rigid 3D deformations of the skin and loose apparel, which are not fully explained by skeletal motion alone. It uses a novel real-time algorithm that minimizes an energy function based on:
- Dense Photometric Alignment: Matching the projected 3D model to the input image colors.
- Dense Silhouette Alignment: Ensuring the model's boundary aligns with the detected silhouette. A novel "body part mask" is used to handle self-occlusions and improve silhouette fitting.
- Material-Dependent Regularization: A key innovation is the use of automatically identified material regions (skin, dress, pants, etc.) on the template to apply different non-rigidity weights. This allows the system to model the distinct deformation behaviors of skin versus loose clothing.
- Temporal Priors: Encouraging smooth velocity and acceleration of non-rigid deformations.
To achieve real-time performance (over 25 Hz), the system utilizes a pipelined architecture across a CPU and two GPUs. Stage I (pose estimation) and Stage II (non-rigid registration) run concurrently on separate GPUs, with the CPU handling initial data processing. The optimization problems are solved using custom data-parallel Gauss-Newton solvers.
Datasets
The paper introduces a new dataset for evaluation, consisting of over 20 minutes of video footage across 11 scenarios featuring various clothing types and challenging motions. This dataset includes semantically segmented, rigged, and textured templates, calibrated camera parameters, and empty background images. For evaluation, they also used existing video sequences and compared their method to prior work on their own sequences and sequences from other papers.
Key Insights and Ideas
- Real-time Monocular Performance Capture: This is the first system to achieve dense, full-body human performance capture in real-time from a single RGB camera.
- Material-Aware Non-Rigid Deformation: Differentiating the regularization weights based on material (skin vs. clothing) is crucial for accurately capturing the distinct deformation behaviors of different body parts and apparel. This is a significant improvement over methods that apply uniform regularization.
- Pipelined GPU Optimization: Effectively distributing the two-stage optimization process across multiple GPUs and the CPU is essential for meeting the real-time constraint.
- Combining Sparse and Dense Cues: The system leverages both sparse (2D/3D joints, facial landmarks) and dense (silhouette, photometric) information, along with a learned pose estimator, to achieve robust and accurate results.
- Body Part Mask for Silhouette Alignment: Using a mask derived from skinning weights to dilate the projected body parts significantly improves silhouette alignment, especially in challenging self-occluded poses.
Key Results
- Speed: The system achieves over 25 FPS on a system with two commodity GPUs.
- Accuracy:
- Silhouette Overlap (IoU): Outperforms existing monocular skeleton-only methods by 8.5-9.4% and is only 4.3% lower than the off-line MonoPerfCap method on a challenging sequence. On another sequence with similar foreground segmentation quality, it achieves comparable IoU to MonoPerfCap.
- Surface Reconstruction Error: Comes very close to off-line multi-view methods and significantly outperforms other monocular non-rigid methods.
- Skeletal Pose Estimation Error: Outperforms other monocular methods, including skeleton-only approaches, demonstrating that the combined surface and skeleton reconstruction benefits pose estimation.
- Qualitative Results: The system produces visually plausible and temporally coherent reconstructions that accurately overlay the input video, even for challenging motions like fast rotations and jumping, and for loose clothing like skirts and dresses.
Limitations and Future Work
- Template Dependency: The accuracy relies on a good initial template acquisition. Errors in the template or its segmentation cannot be fully recovered.
- Topological Changes: The system struggles with significant topological changes (e.g., removing clothes) or deformations along the camera viewing axis that are too far from the template.
- Extreme Poses: The underlying deep pose estimator can fail for extreme poses not seen during training, leading to temporary glitches. While the system often recovers, these are still limitations.
- Segmentation Sensitivity: Performance can be affected by inaccurate foreground/background segmentation due to shadows, similar colors, or changing illumination.
- Occlusions: Extensive or complete occlusions of body parts lead to less accurate estimates compared to multi-view methods, as there is no visual evidence.
- Future Work: The authors suggest exploring joint estimation of facial expressions, hand pose, and hair dynamics from a single monocular camera. They also propose incorporating more sophisticated segmentation methods and addressing shading/illumination effects.
Summary updated