Capture Dense: Markerless Motion Capture Meets Dense Pose Estimation

Xiu Li, Yebin Liu, Hanbyul Joo, Qionghai Dai, Yaser Sheikh

TL;DR

This paper introduces "Capture Dense," a novel framework that tightly integrates markerless motion capture with dense human pose estimation. By creating a symbiotic loop where motion capture data improves dense pose estimation and vice-versa, they achieve more accurate and efficient human pose and shape reconstruction, while also generating a valuable dataset for training better dense pose detectors.

What Problem Does This Paper Address?

Accurately capturing human motion and shape in 3D from images and videos is a long-standing challenge in computer vision and graphics. Traditional markerless motion capture methods often rely on sparse 2D joint detection, which can struggle with accuracy, especially for complex body parts like hands and arms, and are computationally expensive. On the other hand, dense pose estimation, which provides per-pixel semantic information, has shown promise but often suffers from limited and noisy training data, particularly for challenging viewpoints. This paper aims to bridge this gap by creating a unified system that leverages the strengths of both approaches to overcome their individual limitations.

Methods

The core of Capture Dense is an iterative optimization loop that combines markerless motion capture and dense pose estimation.

  1. Markerless Motion Capture with Dense Pose Refinement:
    • They start with existing 2D joint detectors (like OpenPose) to extract multi-view 2D joint positions.
  2. These are used to fit an initial 3D human model (Adam model) by minimizing joint position errors and using anatomical priors.
  3. Crucially, they then refine this initial pose and shape using dense pose information. This involves driving the 3D model's surface vertices to align with multi-view dense pose correspondences (UVI maps). This refinement step significantly improves accuracy, especially for body part rotations and challenging poses.
  4. This approach is noted to be much faster than prior methods like TotalCapture.
  5. Dense Pose Dataset Generation:
    • The high-quality motion capture results from the first stage are used to automatically generate a large-scale, dense pose dataset.
  6. They create an "Adam-UVI map" by aligning the Adam model to the SMPL model (used by DensePose) to enable rendering of dense pose annotations.
  7. This process leverages the accurate 3D shape and pose from their motion capture to create dense pose labels for multi-view images, overcoming the limitations of manual annotation (e.g., ambiguity, corner cases).
  8. Dense Pose Detector Training:
    • The newly generated dataset is used to train a dense pose detector.
  9. The entire process is then iterated: the improved dense pose detector is used in the markerless motion capture stage, leading to further improvements in both components.

Datasets

  • DensePose-COCO: Used for initial training and evaluation of the dense pose detector.
  • Panoptic-Dense: A dataset generated by the authors using their markerless motion capture system. This dataset is described as being more dense, accurate, and consistent than existing datasets, particularly for handling corner cases and unusual viewpoints. They used 30K annotations for training and 8K for testing.

Key Insights and Ideas

  • Symbiotic Relationship: The paper highlights a powerful synergy between motion capture and dense pose estimation. Dense pose provides rich semantic information that significantly aids in accurate and efficient motion capture, while robust motion capture provides the necessary ground truth for generating high-quality dense pose training data.
  • Iterative Refinement: The core innovation lies in the iterative loop. By feeding the improved dense pose detector back into the motion capture pipeline, they achieve a virtuous cycle of improvement for both tasks.
  • Data Generation for Dense Pose: Manually annotating dense pose data is laborious and prone to errors. This work demonstrates a practical method to automatically generate large-scale, high-quality dense pose datasets using existing motion capture systems, which is crucial for advancing dense pose estimation.
  • Addressing Corner Cases: The generated dataset and the refinement process are shown to be particularly effective in handling challenging viewpoints and complex body poses that are difficult for manual annotation or sparse methods.

Key Results

  • Markerless Motion Capture:
    • Achieved comparable or better shape and pose reconstruction accuracy than state-of-the-art methods like TotalCapture.
  • Substantially reduced running time (e.g., ~0.54s per frame for their method vs. 60s+ for TotalCapture's joint fitting and 120s+ for point cloud optimization).
  • Visually demonstrated improved accuracy in arm rotations.
  • Dense Pose Detector:
    • Significantly improved dense pose accuracy across various viewpoints and pose clusters compared to the original DensePose baseline.
  • Achieved higher mAP scores on the Panoptic-Dense testing set.
  • Demonstrated better dense feature consistency among different views.

Limitations and Future Work

  • Limitations:
    • The paper notes that while their method improves upon existing ones, the accuracy of the generated dataset is still limited by the underlying human annotated dataset (DensePose-COCO), which is mostly from front-view.
  • Qualitative evaluation of arm rotations is mentioned as being better than TotalCapture but hard to quantify.
  • The performance improvement of their training scheme on the DensePose-COCO dataset is described as "not substantial," possibly due to the dataset's inherent limitations.
  • Future Work:
    • The authors believe their work can inspire future research in areas like stereo matching and 3D reconstruction on human bodies.
  • They suggest that the improved dense pose detector can be used in interesting tasks such as single-image human body reconstruction.
  • The iterative deep learning framework, which encodes multi-view information into a single dense pose detector, is seen as a promising direction.

Summary updated

Abstract

We present a method to combine markerless motion capture and dense pose feature estimation into a single framework. We demonstrate that dense pose information can help for multiview/single-view motion capture, and multiview motion capture can help the collection of a high-quality dataset for training the dense pose detector. Specifically, we first introduce a novel markerless motion capture method that can take advantage of dense parsing capability provided by the dense pose detector. Thanks to the introduced dense human parsing ability, our method is demonstrated much more efficient, and accurate compared with the available state-of-the-art markerless motion capture approach. Second, we improve the performance of available dense pose detector by using multiview markerless motion capture data. Such dataset is beneficial to dense pose training because they are more dense and accurate and consistent, and can compensate for the corner cases such as unusual viewpoints. We quantitatively demonstrate the improved performance of our dense pose detector over the available DensePose. Our dense pose dataset and detector will be made public.
BibTeX
@misc{li2018capture,
  author = {Xiu Li and Yebin Liu and Hanbyul Joo and Qionghai Dai and Yaser Sheikh},
  title = {{Capture Dense: Markerless Motion Capture Meets Dense Pose Estimation}},
  year = {2018},
  eprint = {1812.01783},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.01783},
}
Figure