Sequential Non-Rigid Structure-from-Motion with the 3D-Implicit Low-Rank Shape Model

Marco Paladini, Adrien Bartoli, Lourdes Agapito

Abstract

Non-Rigid Structure-from-Motion (NRSfM) has traditionally been approached as a batch optimization problem, requiring an entire video sequence to be processed at once. This limitation has hindered its application in real-time scenarios such as augmented reality and surgical navigation. This paper introduces the first sequential, frame-by-frame approach to NRSfM. By utilizing a novel "3D-implicit" low-rank shape model, the proposed method simplifies the motion matrix into a block-diagonal structure. This allows the system to learn new deformation modes dynamically as they occur. Experimental results demonstrate that this sequential pipeline achieves accuracy comparable to state-of-the-art batch methods while maintaining a bounded computational cost per frame.

Introduction

Structure-from-Motion (SfM) is a fundamental computer vision task involving the estimation of 3D structures from 2D image sequences. While rigid SfM is a mature field capable of real-time performance, Non-Rigid Structure-from-Motion (NRSfM)—which deals with deforming objects like human faces or moving cloth—is significantly more complex. Historically, NRSfM has been treated as a batch problem, where optimization occurs only after the full sequence is captured. The authors of this paper aim to bridge this gap, moving NRSfM toward the real-time capabilities of its rigid counterparts.

The 3D-Implicit Low-Rank Shape Model

Traditional NRSfM models typically represent a shape as a linear combination of $K$ basis shapes. This representation results in a motion matrix with a complex, repetitive structure that is difficult to update sequentially.

The authors propose a "3D-implicit" model that rearranges how 3D coordinates are stored. This mathematical shift results in a block-diagonal motion matrix, which is significantly easier to solve. A major advantage of this model is its flexibility regarding "rank." In traditional models, the rank must grow in multiples of three to account for 3D coordinates. In the 3D-implicit model, the rank can grow by one at a time. This allows the system to naturally handle "degenerate" or simple deformations—such as a jaw moving in a single dimension—without the overhead of a full 3D basis update.

Sequential Estimation Pipeline

The proposed method treats NRSfM as a tracking problem with an evolving model. The pipeline consists of three main phases:

  1. Bootstrapping: The system begins by assuming the object is rigid for the initial frames to establish a baseline "mean shape."
  2. Camera Tracking: For every new frame, the system estimates the camera's pose and the current deformation coefficients using a sliding window bundle adjustment. This ensures that the computation time remains constant regardless of the total video length.
  3. Model Update (Rank Growth): The system monitors the reprojection error—the difference between the model's predicted point locations and their actual appearance in the image. If the error exceeds a specific threshold, the system triggers a "rank growth," adding a new deformation mode to the model to account for the new movement.

Experimental Evaluation

The methodology was evaluated on two primary datasets:

  • CMU-Face Sequence: A motion capture (MOCAP) dataset containing 316 frames of a person performing facial expressions. This provided ground truth 3D data for quantitative analysis.
  • Actress Sequence: A 102-frame real-world video of an actress talking, used to demonstrate qualitative performance on natural, unconstrained video.

Key Results and Insights

  • High Accuracy: On the CMU-Face dataset, the sequential method achieved an average 3D error of 2.9%. This is on par with state-of-the-art batch methods like EM-LDS, proving that sequential processing does not require a sacrifice in precision.
  • Model Stability: The system successfully "grew" the model's rank as new expressions were introduced, with the rank typically stabilizing around 12–15 modes for facial sequences.
  • Efficiency: Because the system utilizes a sliding window (focusing only on the last $W$ frames), the computational cost per frame is bounded, making it a viable candidate for real-time applications.
  • Algorithmic Simplicity: The block-diagonal structure of the motion matrix allows for linear initialization of camera poses, which makes the subsequent non-linear optimization (bundle adjustment) much more stable.

Limitations and Future Work

Despite its successes, the algorithm currently assumes that point correspondences (feature tracking) are already established across frames. It does not yet solve the "matching" problem autonomously. Additionally, while the algorithm is sequential, the current MATLAB implementation is not yet "real-time" in practice; the authors suggest that a C++ implementation would be necessary for live use. Future research will focus on integrating feature tracking and 3D modeling into a single, unified process to improve robustness against occlusions.

Summary updated

Abstract

So far the Non-Rigid Structure-from-Motion problem has been tackled using a batch approach. All the frames are processed at once after the video acquisition takes place. In this paper we propose an incremental approach to the estimation of deformable models. Image frames are processed online in a sequential fashion. The shape is initialised to a rigid model from the first few frames. Subsequently, the problem is formulated as a model based camera tracking problem, where the pose of the camera and the mixing coefficients are updated every frame. New modes are added incrementally when the current model cannot model the current frame well enough. We define a criterion based on image reprojection error to decide whether or not the model must be updated after the arrival of a new frame. The new mode is estimated performing bundle adjustment on a window of frames. To represent the shape, we depart from the traditional explicit low-rank shape model and propose a variant that we call the 3D-implicit low-rank shape model. This alternative model results in a simpler formulation of the motion matrix and provides the ability to represent degenerate deformation modes. We illustrate our approach with experiments on motion capture sequences with ground truth 3D data and with real video sequences.
BibTeX
@inproceedings{paladini2010sequential,
  author = {Marco Paladini and Adrien Bartoli and Lourdes Agapito},
  title = {{Sequential Non-Rigid Structure-from-Motion with the 3D-Implicit Low-Rank Shape Model}},
  booktitle = {European Conference on Computer Vision},
  year = {2010},
  doi = {10.1007/978-3-642-15552-9\_2},
}
Figure