A Sequential Learning-Based Approach for Monocular Human Performance Capture

Jianchun Chen, Jayakorn Vongkulbhisal, Fernando De la Torre FradeWACV 2024

TL;DR

This paper introduces a novel sequential learning approach to reconstruct detailed 3D human performance from monocular RGB video. By breaking down the problem into estimating body shape, then consensus clothing, then frame-specific clothing deformation, and finally wrinkles, the method achieves state-of-the-art accuracy while significantly reducing computational cost compared to existing optimization-based techniques.

What Problem Does This Paper Address?

Capturing realistic 3D human performance from everyday videos is a long-standing challenge in computer vision. While existing methods can estimate body shape, they often struggle with fine-grained details like clothing wrinkles and temporal consistency, especially in unconstrained environments. Optimization-based methods can achieve high precision but are computationally expensive and require good initializations. Neural network-based methods are faster but often lack accuracy and robustness to challenging poses or views. This paper aims to bridge this gap by developing a method that is both accurate and efficient, capable of handling monocular video input.

Methods

The core innovation lies in a sequential learning-to-optimize framework that decomposes the complex task of clothing capture into three stages, each handled by a specialized neural network.

  • Gradient Rectification Network (GRN): This is the central component. Instead of directly predicting 3D shape from image features, the GRN takes sparse and noisy gradients (derived from 2D image losses like silhouette or surface normals) as input. It then predicts a dense, rectified gradient that guides an iterative optimization process to update the 3D mesh vertices. This approach is trained on synthetic data but shows strong generalization.
  • Sequential Stages:
    1. Body Pose and Shape Estimation: An initial naked SMPL body model is estimated using existing deep learning methods and then refined.
  • Consensus Clothing Shape Estimation: A single, average clothing shape is estimated that minimizes rendering errors across multiple frames of the video. This provides a robust base for clothing.
  • Frame-dependent Clothing Shape Estimation: A second GRN refines the consensus shape to capture frame-specific clothing deformations, accounting for dynamic movements.
  • Wrinkle Extraction: A final GRN focuses on generating high-frequency wrinkle details by aligning rendered surface normals with target normal maps.

This staged approach allows each network to focus on a specific aspect of the clothing, leading to more accurate and detailed reconstructions.

Datasets

  • Training: The GRNs are trained using synthetic 3D dynamic human motion datasets. For wrinkle extraction, the TailorNet dataset is used, which provides high-resolution SMPL-registered 3D scans.
  • Evaluation:
    • Pablo sequence: A multi-view dataset with reconstructed 3D scans used for quantitative evaluation.
  • Online and smartphone-shot videos: Used for qualitative results on in-the-wild scenarios.
  • BUFF dataset: A synthetic video sequence used for testing robustness under different viewpoints.

Key Insights and Ideas

  • Gradient-based Learning for Shape: The paper argues that learning to predict gradients, rather than direct shape parameters or implicit functions, is more robust for fine-grained mesh refinement, especially when dealing with sparse and noisy 2D cues.
  • Sequential Decomposition: Breaking down the complex clothing capture problem into distinct stages (consensus shape, frame-dependent deformation, wrinkles) significantly simplifies each sub-problem and allows for specialized network designs.
  • Generalization from Synthetic Data: The GRNs, trained on synthetic data, demonstrate surprisingly good generalization to real-world videos, highlighting the effectiveness of the learned gradient rectification approach.
  • Efficiency through GRN: By predicting gradients, the method avoids computationally expensive differentiable renderers in the inference loop, leading to a substantial speedup.

Key Results

  • Accuracy: The proposed method achieves state-of-the-art results, outperforming existing learning-based methods and approaching the performance of optimization-based methods that leverage pre-scanned templates. For instance, on the Pablo sequence, the method achieves a point-to-surface error of 17.4 mm, competitive with top baselines.
  • Speed: The sequential approach significantly reduces runtime. Per-frame runtime is reported to be as low as 1.23 seconds, making it an order of magnitude faster than optimization-based methods. This is attributed to avoiding differentiable renderers and directly predicting a mesh.
  • Robustness: The method shows robustness to challenging poses, novel views, and even inaccurate 2D segmentation results, outperforming methods sensitive to these factors.

Limitations and Future Work

  • Limitations:
    • The method's performance might be limited for highly complex garment types not well-represented by the SMPL+D model (e.g., dresses, hoodies).
  • While robust, the method still relies on the quality of the initial body pose and shape estimation.
  • Future Work: The authors suggest that their sequential learning-to-optimize framework could be extended to other areas of 3D reconstruction and performance capture. Further improvements could focus on handling a wider variety of garment types and achieving even higher levels of detail.

Summary updated

BibTeX
@inproceedings{chen2024sequential,
  author = {Jianchun Chen and Jayakorn Vongkulbhisal and Fernando De la Torre Frade},
  title = {{A Sequential Learning-Based Approach for Monocular Human Performance Capture}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure