Deformable Motion 3D Reconstruction by Union of Regularized Subspaces

Antonio Agudo, Francesc Moreno-NoguerICIP 2018 · OpenReview

TL;DR

This paper introduces a novel method for reconstructing 3D human motion from monocular video by modeling it as a union of regularized subspaces. By incorporating "order-varying temporal regularization," the approach effectively captures both smooth shape deformations and similarities between video frames, leading to more accurate and robust 3D reconstructions and automatic motion segmentation.

What Problem Does This Paper Address?

Reconstructing the 3D shape and motion of non-rigid objects (like humans) from 2D video is a challenging problem due to inherent ambiguities. Traditional methods often rely on low-rank assumptions, which can be too simplistic for complex deformations. While "union of subspaces" models offer more expressiveness, they often fail to leverage the sequential nature of video data. This paper tackles this by proposing a method that explicitly exploits temporal relationships in video to improve both the accuracy of 3D shape reconstruction and the ability to segment motion into distinct action primitives, all in an unsupervised manner.

Methods

The core of the proposed method lies in a novel "order-varying temporal regularization" technique. This approach extends the concept of "union of regularized subspaces" by introducing two key temporal regularization terms:

  • Consecutive Temporal Similarities: This term penalizes deviations between consecutive frames, encouraging smooth motion.
  • Smooth Deformations Over Time: This term directly regularizes the time-varying 3D shape, ensuring that deformations evolve smoothly.

These regularization terms are integrated into an energy-based formulation that is solved using an Augmented Lagrange Multipliers (ALM) algorithm. This unified and unsupervised approach learns all parameters, including camera pose, 3D shape, and motion clusters, without requiring any pre-trained data. The method also handles incomplete 2D trajectories (missing data) through a matrix completion step.

Datasets

The paper evaluates its approach on several human motion capture video sequences, including:

  • An articulated human motion dataset from [4] with five motion activities (Drink, Stretch, Yoga, Pick-up, Dance).
  • An American-sign-language sequence.
  • A "back sequence" with dense 2D trajectories from [13].

The authors do not introduce new datasets but rather use existing ones to demonstrate the effectiveness of their method against state-of-the-art approaches.

Key Insights and Ideas

  • Exploiting Temporal Structure is Crucial: The paper strongly emphasizes that simply using a union of subspaces is insufficient. Explicitly modeling the sequential nature of video data through temporal regularization is key to achieving superior results.
  • "Order-Varying" Regularization: The concept of applying regularization that adapts to the "order" of temporal relationships (e.g., first-order, second-order) allows for flexible and powerful modeling of motion dynamics.
  • Unified and Unsupervised Learning: The ability to jointly learn camera pose, 3D shape, and motion segmentation without prior training data or explicit subspace rank selection is a significant advantage.
  • Robustness to Missing Data: The method demonstrates resilience to incomplete 2D trajectories, a common issue in real-world video data.

Key Results

  • Superior 3D Reconstruction Accuracy: The proposed method consistently outperforms state-of-the-art approaches in terms of 3D reconstruction error, especially on noisy observations. Reductions in 3D error range from 12% to 424% compared to other methods.
  • Accurate Motion Grouping: The algorithm successfully segments motion into distinct action primitives, with the affinity matrix estimation showing high accuracy compared to ground truth.
  • Higher-Order Regularization Benefits: Using higher-order temporal regularization approximations generally leads to more accurate 3D reconstructions.
  • No Subspace Rank Tuning Required: Unlike competing methods, the proposed approach does not require manual tuning of the subspace rank, simplifying its application.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current formulation is based on orthographic projection. While this is a common simplification, it limits the applicability to scenarios where perspective effects are significant. The authors also mention that the "order-varying" regularization is applied to specific temporal relationships, and exploring other forms of temporal dependencies could be beneficial.
  • Future Work: The authors express interest in extending their research to solve the problem in a sequential fashion, processing data as it arrives. This would enable real-time applications and online learning. Further exploration of more complex temporal regularization schemes is also a promising direction.

Summary updated

Abstract

This paper presents an approach to jointly retrieve camera pose, time-varying 3D shape, and automatic clustering based on motion primitives, from incomplete 2D trajectories in a monocular video. We introduce the concept of order-varying temporal regularization in order to exploit video data, that can be indistinctly applied to the 3D shape evolution as well as to the similarities between images. This results in a union of regularized subspaces which effectively encodes the 3D shape deformation. All parameters are learned via augmented Lagrange multipliers, in a unified and unsupervised manner that does not assume any training data at all. Experimental validation is reported on human motion from sparse to dense shapes, providing more robust and accurate solutions than state-of-the-art approaches in terms of 3D reconstruction, while also obtaining motion grouping results.
BibTeX
@inproceedings{agudo2018deformable,
  author = {Antonio Agudo and Francesc Moreno-Noguer},
  title = {{Deformable Motion 3D Reconstruction by Union of Regularized Subspaces}},
  booktitle = {ICIP},
  year = {2018},
  url = {https://openreview.net/forum?id=b3rNJB9jVJ},
}
Figure