Towards Accurate Markerless Human Shape and Pose Estimation over Time

Yinghao Huang, Federica Bogo, Christoph Lassner, Angjoo Kanazawa, Peter V. Gehler, Ijaz Akhter, Michael J. BlackarXiv 2017 · arXiv

This paper presents MuVS (Multi-View SMPLify), a fully automatic framework for estimating accurate 3D human pose and body shape from multi-view (and even monocular) video. By extending the single-image SMPLify method with multi-view constraints, CNN-based silhouette fitting, and a robust temporal prior, the authors achieve state-of-the-art results while producing highly realistic 3D avatars without the need for markers or specialized backgrounds.

TL;DR

MuVS is a fully automatic system that reconstructs realistic 3D human pose and shape from video by fitting a parametric body model to 2D joints and silhouettes. Its key innovation lies in using a generic temporal prior to resolve common errors like "leg swapping" and achieving high-fidelity results that outperform previous markerless motion capture methods.

What Problem Does This Paper Address?

Markerless motion capture (mocap)—the ability to track human movement without specialized suits or markers—is a "holy grail" in computer vision. However, existing methods often struggle with:

  • Restrictive Environments: Many require static cameras and known, simple backgrounds for "background subtraction."
  • Ambiguity: 2D pose estimators often confuse the left and right sides of the body (e.g., swapping legs).
  • Lack of Realism: Many methods output simple skeletons or coarse "blob-like" meshes that don't capture the actual shape of the person.
  • Data Dependency: Some methods require training on the specific actor or motion they are trying to track.

Methods

The authors build upon SMPLify, which fits the SMPL (Skinned Multi-Person Linear) body model to 2D joints detected in a single image. MuVS extends this in four critical ways:

  1. Multi-View Integration: Instead of one camera, the model is fit to 2D features detected across multiple synchronized camera views simultaneously, which drastically reduces depth ambiguity.
  2. Silhouette Fitting: The team uses a deep CNN to segment the person from the background. They then refine the 3D body model by fitting its contours to these estimated silhouettes. This removes the need for traditional background subtraction and significantly improves shape accuracy.
  3. DCT Temporal Prior: To ensure smooth motion over time, they use a Discrete Cosine Transform (DCT) basis. This acts as a generic temporal prior, assuming that human motion is inherently smooth. This helps the system "remember" which leg is which when a 2D detector gets confused.
  4. Two-Stage Optimization:
    • Stage One: A per-frame fitting of the SMPL model to 2D joints and silhouettes across all views.
  5. Stage Two: A temporal refinement stage that optimizes pose parameters across consecutive frames (typically 30-frame units) using the DCT prior.
Figure 2: Automatically estimated 2D joint locations and silhouettes used as input for the fitting process.

Datasets

The method was evaluated on two gold-standard benchmarks for human pose estimation:

  • HumanEva: A dataset with synchronized video and ground-truth mocap data for various subjects and motions (walking, boxing).
  • Human3.6M: The largest public dataset for 3D human pose, featuring 11 subjects and 15 different motion types.
  • YouTube Videos: To test the "in-the-wild" monocular capability, the authors applied MuVS to challenging dance videos from the internet.

Key Insights and Ideas

  • Silhouettes are the Key to Realism: While joints provide the "skeleton," silhouettes provide the "flesh." Fitting to contours makes the resulting 3D mesh look like a specific individual rather than a generic mannequin.
  • Temporal Smoothing Resolves Logical Errors: 2D joint detectors are prone to "leg swaps." By looking at a sequence of frames through a DCT lens, the system can identify these swaps as physically impossible jumps in motion and correct them.
  • Multi-View vs. Monocular: While MuVS shines with 2 or 3 cameras, the temporal prior is powerful enough to enable reasonable 3D reconstruction from a single moving camera (monocular video), even with complex backgrounds.
Figure 1: Given multi-view videos, our method can not only yield more accurate 3D pose estimation results, but also more realistic and natural meshes than the state of the art. The entire process is fully automatic. From up to bottom: example input frames; meshes returned by [39]; meshes generated by our method.

Key Results

  • Accuracy: On the HumanEva dataset, MuVS achieved a mean 3D joint error of 55.52 mm using 3 views in a general setting, significantly outperforming the previous state-of-the-art (Rhodin et al. at 67.3 mm).
  • Shape Fidelity: The average vertex-to-vertex distance for shape estimation was reduced to just 8.1 mm when using silhouettes and temporal priors.
  • Qualitative Superiority: As shown in Figure 1, the meshes generated by MuVS are noticeably more realistic and natural-looking than those from competing methods.
Figure 1: Comparison showing MuVS (bottom) produces more realistic and accurate meshes than previous state-of-the-art methods (middle).

Limitations and Future Work

  • Computational Cost: The process is currently slow. Stage one takes about 70 seconds per frame, and the temporal stage takes about 12 minutes for a 30-frame unit. This makes it unsuitable for real-time applications without further optimization.
  • Complexity: The current model does not handle cluttered backgrounds with multiple people or extreme, highly occluded poses perfectly.
  • Model Detail: Future iterations could incorporate more detailed models for hands, faces, and feet to create even more faithful digital doubles.
Figure 6: Demonstration of the generated mesh sequence from a monocular YouTube video.

Summary updated

Abstract

Existing marker-less motion capture methods often assume known backgrounds, static cameras, and sequence specific motion priors, which narrows its application scenarios. Here we propose a fully automatic method that given multi-view video, estimates 3D human motion and body shape. We take recent SMPLify \cite{bogo2016keep} as the base method, and extend it in several ways. First we fit the body to 2D features detected in multi-view images. Second, we use a CNN method to segment the person in each image and fit the 3D body model to the contours to further improves accuracy. Third we utilize a generic and robust DCT temporal prior to handle the left and right side swapping issue sometimes introduced by the 2D pose estimator. Validation on standard benchmarks shows our results are comparable to the state of the art and also provide a realistic 3D shape avatar. We also demonstrate accurate results on HumanEva and on challenging dance sequences from YouTube in monocular case.
BibTeX
@misc{huang2017towards,
  author = {Yinghao Huang and Federica Bogo and Christoph Lassner and Angjoo Kanazawa and Peter V. Gehler and Ijaz Akhter and Michael J. Black},
  title = {{Towards Accurate Markerless Human Shape and Pose Estimation over Time}},
  year = {2017},
  eprint = {1707.07548},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.07548},
}

Loading related content...

Figure