Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture

Denis Tomè, Matteo Toso, Lourdes Agapito, Chris Russell

Abstract

Estimating 3D human pose from 2D images is a fundamentally ill-posed problem due to depth ambiguity. While traditional marker-based motion capture (MoCap) systems provide high accuracy, they are restricted to controlled studio environments. This paper introduces a multi-stage convolutional neural network (CNN) architecture for markerless 3D motion capture that integrates 3D reasoning directly into the estimation loop. By iteratively refining 2D joint detections using provisional 3D models, the system can "rethink" joint locations and recover from initial misdetections. Our approach achieves state-of-the-art results on the Human3.6M dataset, significantly outperforming both monocular and existing multi-view methods.

Introduction

The transition from marker-based to markerless motion capture is essential for analyzing human movement in natural, "in-the-wild" settings. However, 3D pose estimation remains challenging due to occlusions and the loss of depth information during 2D projection. Existing deep learning methods typically follow a linear pipeline: either regressing 3D coordinates directly from images or "lifting" 2D detections into 3D space as a final step.

A critical limitation of these pipelines is the lack of feedback; the 3D model, which possesses inherent knowledge of human skeletal constraints, cannot inform or correct the 2D detector. If a 2D detector misplaces a joint due to occlusion, a standard system has no mechanism to resolve this inconsistency. We propose a framework where 3D consistency acts as a powerful prior to refine 2D detections iteratively.

Methodology

Our architecture consists of a multi-stage refinement process designed to perform joint 2D and 3D reasoning.

Multi-Stage Architecture

Each stage of the network performs three primary operations:

  1. 2D Joint Prediction: The network generates 2D joint heatmaps for each camera view.
  2. 3D Lifting and Robust Estimation: The 2D points are lifted into a coherent 3D pose. To handle catastrophic failures in 2D detection (outliers), we employ a robust estimator based on the Huber loss. Unlike squared-error approaches, the Huber loss allows the model to ignore incorrect joint estimates that do not fit the global 3D structure.
  3. 3D-to-2D Feedback Loop: The estimated 3D pose is projected back into the 2D image space of each camera. These projections are converted into heatmaps and fused with the original 2D predictions, providing a refined starting point for the subsequent stage.
The multi-stage architecture featuring 3D-to-2D feedback loops for iterative refinement.

Camera Modeling

The system utilizes a scaled orthographic camera model. While simpler than full perspective models, it provides computational efficiency and stability. To eliminate temporal artifacts like "flickering" in video, the model marginalizes over possible ground-plane rotations.

Experimental Setup

We evaluated our method primarily on the Human3.6M dataset, which contains millions of frames across various activities captured by four cameras and a ground-truth Vicon system.

Furthermore, we explored data bootstrapping. By using our multi-camera system as an "auto-labeller" on unlabeled subsets of Human3.6M, we generated "silver-standard" labels. These were used to augment the training of standard monocular networks, demonstrating that multi-view consistency can improve single-camera performance without architectural changes.

Results

The proposed method established new performance benchmarks on Human3.6M at the time of publication.

Quantitative Performance

Under Protocol 1 (Mean Per-Joint Position Error), our system achieved an error of 52.8mm, a significant improvement over the previous state-of-the-art multi-camera method by Pavlakos et al. Under Protocol 2 (Error after Procrustes Alignment), we achieved 44.6mm using a 17-joint model.

Comparative results on the Human3.6M dataset under Protocol 2, showing state-of-the-art performance.

Monocular Model Enhancement

By leveraging the multi-view system to generate additional training data, we improved the accuracy of existing monocular models (e.g., Martinez et al.) by approximately 3-4%. This confirms that 3D-informed labels are superior to standard 2D-only training for monocular tasks.

Discussion and Conclusion

While our method demonstrates significant gains, some limitations remain. The use of a scaled orthographic projection introduces a theoretical error floor of approximately 27.9mm due to the neglect of perspective effects. Additionally, the computational cost of a multi-stage refinement loop across multiple views poses challenges for real-time deployment.

Future work will focus on extending this "rethinking" paradigm to full perspective camera models and more diverse, unconstrained multi-view datasets to move closer to truly "in-the-wild" markerless motion capture.

Summary updated

BibTeX
@inproceedings{tome2018rethinking,
  author = {Denis Tomè and Matteo Toso and Lourdes Agapito and Chris Russell},
  title = {{Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture}},
  booktitle = {3DV},
  year = {2018},
  doi = {10.1109/3dv.2018.00061},
}
Figure