Capturing Complex 3D Human Motions with Kernelized Low-Rank Representation from Monocular RGB Camera

Xuan Wang, Fei Wang, Yanan ChenSensors 2017 · DOI

TL;DR

This paper introduces a novel Non-Rigid Structure from Motion (NRSfM) method that leverages kernelized low-rank representation to capture complex 3D human motions from monocular RGB cameras. By assuming that 3D human shapes lie on a union of nonlinear subspaces, the method achieves more accurate reconstructions than previous linear models, and can be extended to marker-less pose estimation using CNN-based joint detectors.

What Problem Does This Paper Address?

Recovering the 3D structure and motion of humans from a single RGB camera (monocular vision) is a fundamentally challenging problem. This is because a 2D image can be the projection of infinitely many different 3D configurations, leading to ambiguity. While previous methods have tackled this by introducing "priors" – assumptions about the nature of human motion – these often rely on linear models that struggle with the complex, non-linear deformations humans exhibit. This paper aims to bridge this gap by proposing a method that can better model these complex, non-linear motions, and also extend this capability to real-world scenarios where markers are not used.

Methods

The core of the proposed method is a Non-Rigid Structure from Motion (NRSfM) framework enhanced with Kernelized Low-Rank Representation (KLRR).

  • NRSfM Foundation: The method builds upon existing NRSfM techniques that aim to reconstruct 3D shapes and motions from a sequence of 2D observations.
  • Low-Rank Assumption: A key prior is that the 3D shapes across different frames can be represented by a low-dimensional subspace. Traditional methods often assume a single linear subspace.
  • Kernelized Low-Rank Representation (KLRR): This paper's innovation is to move beyond linear subspaces. KLRR maps the data into a higher-dimensional feature space where nonlinear relationships can be captured as linear ones. This allows for a more flexible and accurate modeling of complex human deformations.
  • Soft Inextensibility Constraint: To further improve accuracy, a "soft inextensibility" constraint is incorporated. This prior assumes that the lengths between connected joints (like an elbow and wrist) should remain relatively constant, preventing unrealistic stretching or shrinking.
  • Marker-less Extension: The authors demonstrate how to adapt their KLRR-NRSfM method for marker-less 3D human pose estimation. This is achieved by integrating a Convolutional Neural Network (CNN) based 2D human joint detector. Instead of relying on precise 2D point tracks from markers, the CNN provides "heightmaps" indicating the likelihood of a joint's 2D location, which the framework then uses.
  • Optimization: An Augmented Lagrangian Multiplier (ALM) algorithm is used to solve the complex optimization problem formulated by the KLRR-NRSfM framework.

Datasets

The paper evaluates its methods on several datasets:

  • CMU Motion Capture (MoCap) dataset: Used for generating synthetic 2D projections for evaluating the marker-based method.
  • Utrecht Multi-Person Motion (UMPM) benchmark: Used for evaluating the marker-based method on real-world 2D projections.
  • Human3.6M dataset: Used for evaluating the marker-less extension of the method.

Key Insights and Ideas

  • Non-linear Subspaces are Crucial for Human Motion: The paper strongly suggests that human motion is inherently non-linear, and that modeling it with linear subspaces is a significant limitation. KLRR's ability to capture these non-linear relationships is a key insight.
  • KLRR Outperforms LRR in Temporal Consistency: The subspace analysis (Figure 2) visually demonstrates that KLRR leads to more temporally consistent motion clustering compared to standard Low-Rank Representation (LRR), which is crucial for smooth motion recovery.
  • Soft Inextensibility is a Powerful Prior: The inclusion of the soft inextensibility constraint further refines the reconstruction by enforcing realistic human body constraints.
  • CNNs Enable Practical Marker-less 3D Pose: The successful integration of CNN-based 2D joint detectors shows a practical path towards marker-less 3D human pose estimation, making the technology more accessible for real-world applications.

Key Results

  • Improved Accuracy: The proposed TUNS (Temporal Union of Nonlinear Subspaces) method, particularly the KLRR-based version, consistently outperforms several state-of-the-art NRSfM baselines on both synthetic and real-world datasets in terms of mean and median reconstruction errors. For example, in Table 1, TUNS achieves the best or second-best results across most sequences.
  • Effective Marker-less Estimation: The marker-less version, leveraging CNN detectors, demonstrates accurate 3D pose reconstructions on the Human3.6M dataset, showing its viability for practical scenarios.
  • Robustness to Complex Motions: The KLRR approach is shown to be more suitable for modeling complex deformations, leading to more accurate reconstructions compared to linear methods.

Limitations and Future Work

  • Reconstructability Problem: Similar to other NRSfM techniques, the method's performance degrades when the input sequence has a limited range of viewpoints or when the human body appears very small. More sophisticated priors might be needed in such cases.
  • Camera Rotation Estimation: The paper acknowledges that camera rotation estimation is not explicitly handled and suggests it as an open problem for future research, especially under the union of nonlinear subspaces assumption.
  • Occlusion Handling: The paper notes that handling occlusions (when joints are hidden) is a challenge, particularly for the marker-less approach. While CNN detectors are more robust than traditional tracking, an explicit mechanism for occlusion handling is suggested for future work.

Summary updated

Abstract

Recovering 3D structures from the monocular image sequence is an inherently ambiguous problem that has attracted considerable attention from several research communities. To resolve the ambiguities, a variety of additional priors, such as low-rank shape basis, have been proposed. In this paper, we make two contributions. First, we introduce an assumption that 3D structures lie on the union of nonlinear subspaces. Based on this assumption, we propose a Non-Rigid Structure from Motion (NRSfM) method with kernelized low-rank representation. To be specific, we utilize the soft-inextensibility constraint to accurately recover 3D human motions. Second, we extend this NRSfM method to the marker-less 3D human pose estimation problem by combining with Convolutional Neural Network (CNN) based 2D human joint detectors. To evaluate the performance of our methods, we apply our marker-based method on several sequences from Utrecht Multi-Person Motion (UMPM) benchmark and CMU MoCap datasets, and then apply the marker-less method on the Human3.6M datasets. The experiments demonstrate that the kernelized low-rank representation is more suitable for modeling the complex deformation and the method consequently yields more accurate reconstructions. Benefiting from the CNN-based detector, the marker-less approach can be applied to more real-life applications.
BibTeX
@inproceedings{wang2017capturing,
  author = {Xuan Wang and Fei Wang and Yanan Chen},
  title = {{Capturing Complex 3D Human Motions with Kernelized Low-Rank Representation from Monocular RGB Camera}},
  booktitle = {Sensors},
  year = {2017},
  doi = {10.3390/s17092019},
}
Figure