Deep Non-Rigid Structure From Motion With Missing Data

Chen Kong, Simon LuceyIEEE TPAMI 2021 · OpenReview

TL;DR

This paper introduces a novel deep learning approach for Non-Rigid Structure from Motion (NRSfM) that leverages hierarchical sparse coding to reconstruct 3D shapes from 2D correspondences, even with missing data. This method significantly advances the state-of-the-art by handling unprecedented scale and shape complexity without requiring 3D supervision.

What Problem Does This Paper Address?

Reconstructing the 3D structure and camera poses of non-rigid objects from a series of 2D images (Non-Rigid Structure from Motion or NRSfM) is a fundamental problem in computer vision. Existing NRSfM algorithms face limitations in the number of images they can process and the complexity of shape variations they can handle. These limitations often stem from the inherent difficulty in modeling the degrees of freedom involved in non-rigid deformation. Furthermore, real-world scenarios frequently involve missing or occluded 2D points, which traditional methods struggle with, often requiring complex matrix completion techniques. This paper aims to overcome these limitations, enabling NRSfM to be applied to more challenging and practical vision problems.

Methods

The core innovation lies in a novel hierarchical sparse coding model for NRSfM. This model represents the 3D shape as a hierarchical composition of sparse codes, allowing for more complex and stable representations than traditional low-rank or single-layer sparse coding. This hierarchical prior is then translated into an unsupervised deep neural network (DNN) auto-encoder.

The network architecture consists of:

  • Encoder: This part takes 2D correspondences (W) and maps them to a hidden hierarchical block sparse code (ΨN) through a series of layers that mimic single-iteration block ISTA (Iterative Shrinkage and Thresholding Algorithm). Crucially, the learnable parameters (bi) in these layers allow the network to learn the optimal weighting for shape regularization, eliminating the need for cumbersome grid searches.
  • Bottleneck: This layer disentangles the pose (camera M) from the shape representation. The authors propose a novel relaxation that decouples the estimation of the camera and the sparse hidden code, improving robustness and computational efficiency.
  • Decoder: This part reconstructs the 3D shape (S) from the learned sparse code and hierarchical dictionaries, using a similar soft-thresholding mechanism as the encoder to enforce sparsity.

A key aspect is the network's ability to handle missing data by incorporating a mask directly into the first layer's objective function, avoiding the need for explicit matrix completion. The authors also adapt the model for weak perspective projection, which is more suitable for many real-world scenarios than orthogonal projection.

Datasets

The paper evaluates its method on several datasets:

  • IKEA Furniture Dataset: Used for evaluating performance on object categories like beds, chairs, sofas, and tables.
  • PASCAL3D+ Dataset: A larger dataset with twelve object categories, used to test performance under various conditions (orthogonal/weak perspective projection, complete/missing measurements, noise).
  • CMU Motion Capture Dataset: A large-scale dataset used to evaluate the method's performance on sequences with thousands of frames and unseen motions.

Key Insights and Ideas

  • Hierarchical Sparse Coding for Stability: The key insight is that a hierarchical structure in sparse coding, rather than a flat one, leads to a more constrained and stable recovery process, even with over-complete dictionaries. This is counter-intuitive, as more layers might suggest more complexity, but here it leads to better regularization.
  • DNN as Interpretable Sparse Coding Solver: The paper provides a strong theoretical link between feed-forward DNNs and hierarchical sparse coding, interpreting the network's layers as iterative steps of a sparse coding algorithm. This offers a degree of interpretability to the "black box" nature of deep networks.
  • Learning Regularization Weights: Learning the regularization weights (e.g., sparsity parameters) directly within the network, rather than through manual tuning, significantly improves efficiency and robustness.
  • Handling Missing Data without Matrix Completion: The proposed method's ability to directly handle missing 2D points without relying on computationally expensive matrix completion is a significant practical advantage.
  • Coherence as a Quality Measure: The paper proposes using the mutual coherence of the final dictionary as a proxy for model quality, enabling over-fitting detection and model selection even without 3D ground truth.

Key Results

  • State-of-the-Art Performance: The proposed method consistently outperforms existing state-of-the-art NRSfM algorithms across various benchmarks and configurations, often by an order of magnitude in terms of normalized mean 3D error. For instance, on the IKEA dataset, the average relative error is 1.00, significantly better than the next best at 11.13.
  • Robustness: The method demonstrates remarkable robustness to noise and missing data. Even with up to 20% of points missing, the performance remains superior to baselines with no missing points.
  • Scalability: The deep learning approach allows for processing large-scale image sequences with hundreds of thousands of images, a feat not achievable by many prior methods.
  • Generalization: The model shows good generalization to unseen motions in the CMU motion capture dataset, indicating its potential for real-world applications.

Limitations and Future Work

  • Weak Perspective Approximation: While an improvement over orthogonal projection, weak perspective is still an approximation of full perspective projection. The authors suggest that exploring full perspective projection would be a valuable next step.
  • Recurrent Networks for Non-linearity: The authors believe recurrent neural networks might be more suitable for handling the non-linearity of perspective projection and potentially for dealing with larger amounts of missing data.
  • Dual Space Reconstruction: The paper suggests that the core ideas could be extended to dual space and trajectory reconstruction, with the introduction of convolutions to the hierarchical sparse model being an interesting avenue.

Summary updated

Abstract

Non-rigid structure from motion (NRSfM) refers to the problem of reconstructing cameras and the 3D point cloud of a non-rigid object from an ensemble of images with 2D correspondences. Current NRSfM algorithms are limited from two perspectives: (i) the number of images, and (ii) the type of shape variability they can handle. These difficulties stem from the inherent conflict between the condition of the system and the degrees of freedom needing to be modeled – which has hampered its practical utility for many applications within vision. In this paper we propose a novel hierarchical sparse coding model for NRSFM which can overcome (i) and (ii) to such an extent, that NRSFM can be applied to problems in vision previously thought too ill posed. Our approach is realized in practice as the training of an unsupervised deep neural network (DNN) auto-encoder with a unique architecture that is able to disentangle pose from 3D structure. Using modern deep learning computational platforms allows us to solve NRSfM problems at an unprecedented scale and shape complexity. Our approach has no 3D supervision, relying solely on 2D point correspondences. Further, our approach is also able to handle missing/occluded 2D points without the need for matrix completion. Extensive experiments demonstrate the impressive performance of our approach where we exhibit superior precision and robustness against all available state-of-the-art works in some instances by an order of magnitude. We further propose a new quality measure (based on the network weights) which circumvents the need for 3D ground-truth to ascertain the confidence we have in the reconstructability. We believe our work to be a significant advance over state-of-the-art in NRSFM.
BibTeX
@article{kong2021deep,
  author = {Chen Kong and Simon Lucey},
  title = {{Deep Non-Rigid Structure From Motion With Missing Data}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=PCv9EQodeF},
}
Figure