Abstract
Non-Rigid Structure-from-Motion (NRSfM) has traditionally been approached as a batch optimization problem, requiring an entire video sequence to be processed at once. This limitation has hindered its application in real-time scenarios such as augmented reality and surgical navigation. This paper introduces the first sequential, frame-by-frame approach to NRSfM. By utilizing a novel "3D-implicit" low-rank shape model, the proposed method simplifies the motion matrix into a block-diagonal structure. This allows the system to learn new deformation modes dynamically as they occur. Experimental results demonstrate that this sequential pipeline achieves accuracy comparable to state-of-the-art batch methods while maintaining a bounded computational cost per frame.
Introduction
Structure-from-Motion (SfM) is a fundamental computer vision task involving the estimation of 3D structures from 2D image sequences. While rigid SfM is a mature field capable of real-time performance, Non-Rigid Structure-from-Motion (NRSfM)—which deals with deforming objects like human faces or moving cloth—is significantly more complex. Historically, NRSfM has been treated as a batch problem, where optimization occurs only after the full sequence is captured. The authors of this paper aim to bridge this gap, moving NRSfM toward the real-time capabilities of its rigid counterparts.
The 3D-Implicit Low-Rank Shape Model
Traditional NRSfM models typically represent a shape as a linear combination of $K$ basis shapes. This representation results in a motion matrix with a complex, repetitive structure that is difficult to update sequentially.
The authors propose a "3D-implicit" model that rearranges how 3D coordinates are stored. This mathematical shift results in a block-diagonal motion matrix, which is significantly easier to solve. A major advantage of this model is its flexibility regarding "rank." In traditional models, the rank must grow in multiples of three to account for 3D coordinates. In the 3D-implicit model, the rank can grow by one at a time. This allows the system to naturally handle "degenerate" or simple deformations—such as a jaw moving in a single dimension—without the overhead of a full 3D basis update.
Sequential Estimation Pipeline
The proposed method treats NRSfM as a tracking problem with an evolving model. The pipeline consists of three main phases:
- Bootstrapping: The system begins by assuming the object is rigid for the initial frames to establish a baseline "mean shape."
- Camera Tracking: For every new frame, the system estimates the camera's pose and the current deformation coefficients using a sliding window bundle adjustment. This ensures that the computation time remains constant regardless of the total video length.
- Model Update (Rank Growth): The system monitors the reprojection error—the difference between the model's predicted point locations and their actual appearance in the image. If the error exceeds a specific threshold, the system triggers a "rank growth," adding a new deformation mode to the model to account for the new movement.
Experimental Evaluation
The methodology was evaluated on two primary datasets:
- CMU-Face Sequence: A motion capture (MOCAP) dataset containing 316 frames of a person performing facial expressions. This provided ground truth 3D data for quantitative analysis.
- Actress Sequence: A 102-frame real-world video of an actress talking, used to demonstrate qualitative performance on natural, unconstrained video.
Key Results and Insights
- High Accuracy: On the CMU-Face dataset, the sequential method achieved an average 3D error of 2.9%. This is on par with state-of-the-art batch methods like EM-LDS, proving that sequential processing does not require a sacrifice in precision.
- Model Stability: The system successfully "grew" the model's rank as new expressions were introduced, with the rank typically stabilizing around 12–15 modes for facial sequences.
- Efficiency: Because the system utilizes a sliding window (focusing only on the last $W$ frames), the computational cost per frame is bounded, making it a viable candidate for real-time applications.
- Algorithmic Simplicity: The block-diagonal structure of the motion matrix allows for linear initialization of camera poses, which makes the subsequent non-linear optimization (bundle adjustment) much more stable.
Limitations and Future Work
Despite its successes, the algorithm currently assumes that point correspondences (feature tracking) are already established across frames. It does not yet solve the "matching" problem autonomously. Additionally, while the algorithm is sequential, the current MATLAB implementation is not yet "real-time" in practice; the authors suggest that a C++ implementation would be necessary for live use. Future research will focus on integrating feature tracking and 3D modeling into a single, unified process to improve robustness against occlusions.
Summary updated