Abstract
Reliable 3D reconstruction often relies on calibrated stereo pairs, which provide depth via triangulation. However, wide baselines complicate feature matching. Conversely, monocular temporal tracking is easier but struggles with depth estimation for non-rigid structures unless significant rigid motion occurs. This paper introduces a "stereo-motion" framework that integrates the advantages of both approaches. By extending the factorization framework to a multi-camera motion model, we enable the reconstruction of time-varying shapes even in the absence of significant rigid motion.
Introduction
In the field of computer vision, recovering the 3D structure of deforming objects remains a significant challenge. Traditional methods generally bifurcate into two categories: Stereo Vision, which uses simultaneous views from multiple cameras, and Structure from Motion (SfM), which tracks features from a single moving camera over time.
While stereo vision offers instantaneous depth, it is often plagued by the "correspondence problem"—the difficulty of matching features across widely spaced views. SfM simplifies feature tracking due to small temporal displacements but typically assumes object rigidity. When applied to non-rigid objects, such as a human face, monocular SfM requires the object to undergo substantial rigid rotation to resolve depth. Without this motion, reconstructions often collapse into flat, inaccurate representations. This paper proposes a "stereo-motion" approach that fuses these methodologies to reconstruct non-rigid 3D shapes effectively.
The Stereo-Motion Framework
The proposed method builds upon the Tomasi-Kanade factorization algorithm. In a standard rigid scenario, a measurement matrix $W$, containing the image coordinates of points across frames, is decomposed into a motion matrix $M$ and a shape matrix $S$ ($W = MS$).
Modeling Non-Rigidity
To account for deformation, the authors utilize a Linear Basis Shape model. In this model, a deforming 3D shape is represented as a weighted linear combination of $D$ basis shapes. For instance, a face's configuration at any moment is treated as a blend of a neutral pose and various deformation states, such as smiling or raising eyebrows.
Stereo Factorization
The core innovation lies in the simultaneous processing of data from a fixed stereo rig. By stacking the image coordinates from both the left and right cameras into a single, unified measurement matrix, the framework enforces a critical constraint: both cameras must observe the same 3D structure and the same deformation weights at any given time step. This shared constraint stabilizes the mathematical solution and provides the "instantaneous baseline" necessary to recover depth even when the object is otherwise stationary.
Optimization Strategy
The reconstruction process follows a two-step optimization pipeline:
- Linear Solution: An initial estimate of the 3D motion and shape is obtained using Singular Value Decomposition (SVD) and metric constraints (specifically, the orthogonality of rotation matrices).
- Non-Linear Bundle Adjustment: This initial estimate is refined by minimizing the reprojection error—the distance between the observed image coordinates and the projected coordinates of the 3D model. The authors employ a Levenberg-Marquardt scheme that exploits the sparse structure of the Jacobian matrix. Because a point's position in one frame is independent of its position in distant frames, the resulting system of equations is sparse, allowing for efficient computation even with large datasets.
Experimental Results
The framework was validated through a series of synthetic and real-world experiments.
Synthetic Validation
Using a Synthetic Cube of random points, the authors demonstrated that while monocular methods fail (reaching error rates of 50%) when rigid motion is low, the stereo-motion approach maintains high accuracy. Tests on a Synthetic Face (160 points, 18 muscle-based deformations) further confirmed that the algorithm could accurately recover complex, non-rigid ground truth data.
Real-World Sequences
The method was applied to two real-world sequences of a human face using physical markers for tracking:
- SMILE sequence: 82 frames capturing a subject smiling.
- EYEBROW sequence: 115 frames of eyebrow movement.
In the "EYEBROW" sequence, where the subject remained relatively still, monocular reconstruction failed to capture the depth of the face, resulting in a flat model. In contrast, the stereo-motion framework successfully recovered the 3D "relief" of the face, including the curvature of the forehead and nose. Furthermore, the non-linear refinement step proved essential for capturing subtle details, such as the specific contours of the lips during a smile.
Conclusion and Future Work
The stereo-motion framework effectively bridges the gap between stereo vision and monocular SfM, providing a robust solution for non-rigid 3D reconstruction. By leveraging the constraints of a stereo rig, the method eliminates the requirement for significant rigid motion.
Future research directions include improving robustness to camera synchronization offsets, extending the model to handle non-fixed camera configurations, and moving toward markerless feature tracking. Additionally, scaling the framework to three or more cameras could further improve accuracy and occlusion handling.
Summary updated