Abstract
Structure from motion is an important theme in computer vision. Although great progress has been made both in theory and applications, most of the algorithms only work for static scenes and rigid objects. In recent years, structure and motion recovery of nonrigid objects and dynamic scenes has received a lot of attention. In this paper, the state-of-the-art techniques for structure and motion factorization of nonrigid objects are reviewed and discussed. First, an introduction of the structure from motion problem is presented, followed by a general formulation of nonrigid structure from motion. Then, an augmented affined factorization framework, by using homogeneous representation, is presented to solve the registration issue in the presence of outlying and missing data. Third, based on the observation that the reprojection residuals of outliers are significantly larger than those of inliers, a robust factorization strategy with outlier rejection is proposed by means of the reprojection residuals, followed by some comparative experimental evaluations. Finally, some future research topics in nonrigid structure from motion are discussed.
Introduction
Structure from Motion (SfM) is a cornerstone of computer vision, enabling the transformation of 2D image sequences into 3D models. While traditional SfM is highly effective for static environments and rigid objects, real-world applications often involve objects that bend, stretch, and move—such as human faces, animals, or inflating balloons. This domain is known as Non-Rigid Structure from Motion (NRSfM).
The primary challenge in NRSfM is that the problem is inherently under-constrained; there are infinite ways a 2D point could move in 3D if the object can change shape arbitrarily. To address this, researchers typically assume a shape is a linear combination of a few "basis shapes." However, a significant practical hurdle remains: data corruption. In real-world video, features are often occluded (missing data) or tracked incorrectly (outliers). Standard algorithms require "registering" images to their center of mass (centroid) before processing. If even a few points are missing or incorrect, the centroid calculation fails, causing the entire 3D reconstruction to become biased or collapse.
Methodology: Robust Factorization
To overcome these challenges, a two-pronged solution is proposed to make NRSfM more resilient to tracking errors and missing data.
Augmented Affine Factorization
Instead of the traditional rank-$3k$ factorization (where $k$ is the number of basis shapes), which requires pre-subtracting the image centroid, this framework introduces a rank-$(3k+1)$ formulation. By utilizing homogeneous coordinates, the translation of the object is folded directly into the matrix factorization. This eliminates the need for manual registration, allowing the algorithm to work directly with "dirty" tracking data.
Residual-Based Outlier Rejection
The framework observes that even when a factorization is performed on corrupted data, the "reprojection residuals" (the distance between the model's predicted point and the actual image point) follow a predictable pattern. Inliers typically follow a Gaussian distribution, while outliers exhibit significantly larger residuals. The algorithm follows an iterative loop:
- Perform an initial factorization using the augmented model.
- Calculate residuals and set a threshold to identify outliers.
- Eliminate outliers and re-estimate the model using a weighted factorization.
- Upgrade the final result from a generic affine space to a metric (Euclidean) 3D space using rotation constraints.
Experimental Evaluation
The method was evaluated using a mix of controlled simulations and classic computer vision benchmarks, including a deformable space cube, a dinosaur sequence, and the FGnet face sequence.
Key Results
The proposed method was compared against "Direct" factorization (no rejection) and other robust schemes such as the Huynh outlier correction and Ke L1-norm minimization:
- Accuracy: In synthetic tests with 20% outliers, the proposed method maintained a low mean reprojection variance, while the "Direct" method's error spiked significantly.
- Real-World Performance: On the dinosaur sequence, the proposed method achieved a mean reprojection error of 0.597, significantly lower than Huynh (0.926) and Ke (0.733).
- Visual Fidelity: The 3D models generated for the face and dinosaur sequences showed smooth, realistic deformations even when the input tracking data was intentionally corrupted.
Conclusion and Future Work
By moving to a rank-$(3k+1)$ model, the mathematical framework becomes inherently more robust to the errors it seeks to solve. The most significant insight is that the preprocessing step of centroid subtraction is often the "Achilles' heel" of NRSfM.
While the current framework is designed for single non-rigid objects using an affine camera model, future research aims to develop a unified algorithm for "General SfM." This would involve automatically segmenting and reconstructing complex scenes containing a mix of rigid, non-rigid, and articulated (jointed) objects in a single pass.
Summary updated