Non-rigid Stereo-motion

A.p. del, Lourdes AgapitoOpenalex Unknown 2007 · DOI · OpenAlex

Abstract

Reliable 3D reconstruction often relies on calibrated stereo pairs, which provide depth via triangulation. However, wide baselines complicate feature matching. Conversely, monocular temporal tracking is easier but struggles with depth estimation for non-rigid structures unless significant rigid motion occurs. This paper introduces a "stereo-motion" framework that integrates the advantages of both approaches. By extending the factorization framework to a multi-camera motion model, we enable the reconstruction of time-varying shapes even in the absence of significant rigid motion.

Introduction

In the field of computer vision, recovering the 3D structure of deforming objects remains a significant challenge. Traditional methods generally bifurcate into two categories: Stereo Vision, which uses simultaneous views from multiple cameras, and Structure from Motion (SfM), which tracks features from a single moving camera over time.

While stereo vision offers instantaneous depth, it is often plagued by the "correspondence problem"—the difficulty of matching features across widely spaced views. SfM simplifies feature tracking due to small temporal displacements but typically assumes object rigidity. When applied to non-rigid objects, such as a human face, monocular SfM requires the object to undergo substantial rigid rotation to resolve depth. Without this motion, reconstructions often collapse into flat, inaccurate representations. This paper proposes a "stereo-motion" approach that fuses these methodologies to reconstruct non-rigid 3D shapes effectively.

The Stereo-Motion Framework

The proposed method builds upon the Tomasi-Kanade factorization algorithm. In a standard rigid scenario, a measurement matrix $W$, containing the image coordinates of points across frames, is decomposed into a motion matrix $M$ and a shape matrix $S$ ($W = MS$).

Modeling Non-Rigidity

To account for deformation, the authors utilize a Linear Basis Shape model. In this model, a deforming 3D shape is represented as a weighted linear combination of $D$ basis shapes. For instance, a face's configuration at any moment is treated as a blend of a neutral pose and various deformation states, such as smiling or raising eyebrows.

Stereo Factorization

The core innovation lies in the simultaneous processing of data from a fixed stereo rig. By stacking the image coordinates from both the left and right cameras into a single, unified measurement matrix, the framework enforces a critical constraint: both cameras must observe the same 3D structure and the same deformation weights at any given time step. This shared constraint stabilizes the mathematical solution and provides the "instantaneous baseline" necessary to recover depth even when the object is otherwise stationary.

Optimization Strategy

The reconstruction process follows a two-step optimization pipeline:

  1. Linear Solution: An initial estimate of the 3D motion and shape is obtained using Singular Value Decomposition (SVD) and metric constraints (specifically, the orthogonality of rotation matrices).
  2. Non-Linear Bundle Adjustment: This initial estimate is refined by minimizing the reprojection error—the distance between the observed image coordinates and the projected coordinates of the 3D model. The authors employ a Levenberg-Marquardt scheme that exploits the sparse structure of the Jacobian matrix. Because a point's position in one frame is independent of its position in distant frames, the resulting system of equations is sparse, allowing for efficient computation even with large datasets.

Experimental Results

The framework was validated through a series of synthetic and real-world experiments.

Synthetic Validation

Using a Synthetic Cube of random points, the authors demonstrated that while monocular methods fail (reaching error rates of 50%) when rigid motion is low, the stereo-motion approach maintains high accuracy. Tests on a Synthetic Face (160 points, 18 muscle-based deformations) further confirmed that the algorithm could accurately recover complex, non-rigid ground truth data.

Real-World Sequences

The method was applied to two real-world sequences of a human face using physical markers for tracking:

  • SMILE sequence: 82 frames capturing a subject smiling.
  • EYEBROW sequence: 115 frames of eyebrow movement.

In the "EYEBROW" sequence, where the subject remained relatively still, monocular reconstruction failed to capture the depth of the face, resulting in a flat model. In contrast, the stereo-motion framework successfully recovered the 3D "relief" of the face, including the curvature of the forehead and nose. Furthermore, the non-linear refinement step proved essential for capturing subtle details, such as the specific contours of the lips during a smile.

Conclusion and Future Work

The stereo-motion framework effectively bridges the gap between stereo vision and monocular SfM, providing a robust solution for non-rigid 3D reconstruction. By leveraging the constraints of a stereo rig, the method eliminates the requirement for significant rigid motion.

Future research directions include improving robustness to camera synchronization offsets, extending the model to handle non-fixed camera configurations, and moving toward markerless feature tracking. Additionally, scaling the framework to three or more cameras could further improve accuracy and occlusion handling.

Summary updated

Abstract

Using a calibrated stereo pair is a common and practical solution to obtain reliable 3-D reconstructions. In its simpler formulation, once the stereo rig is calibrated, the depth of points in the image is estimated by applying triangulation In order to obtain accurate depth estimates, the cameras are usually separated from each other by a significant baseline thus creating widely spaced observations of the same object. The disadvantage of this configuration though, is that having a wide baseline makes the matching of features between pairs of views a more challenging problem. On the other hand, the task of computing temporal tracks from single camera sequences is relatively easier since the images are closely spaced in time. As a drawback, disparities may be insufficient to obtain a reliable depth estimation and, as a result, longer sequences are needed to infer the 3-D structure. Particularly, in the case of non-rigid structure, a sufficient overall rigid motion is necessary to allow the algorithms to estimate the reconstruction parameters correctly. Hence, a question of relevant interest is the feasibility of an approach that efficiently fuses the positive aspects of both methods. The problem of recovering 3-D structure using a stereo-rig moving in time or a stereo rig looking at a moving object has been defined for the rigid case as the stereo-motion problem (Waxman & Duncan, 1998; Following a similar direction, we introduce a multi-camera motion model that is able to deal with a timevarying shape and we present a linear solution based on the factorization framework that is subsequently optimized with a non-linear procedure. Schematically, the chapter is structured as follows. The inference of 3-D structure from an image sequence (single camera case) is introduced in Section 2, focusing particularly on the case of a deforming body. The next section will show how the presented framework, based on a factorization solution of the problem, can be consistently extended to the case of multiple cameras viewing a deforming body and a linear solution to the problem will be as well provided. Section 3 introduces a non-linear optimization strategy to refine the linear solution obtained with the previous method and Section 4 will validate the presented approaches with experimental tests on synthetic and real deforming bodies. Finally we present further considerations over the presented framework and its future extensions.
BibTeX
@inproceedings{del2007non,
  author = {A.P. del and Lourdes Agapito},
  title = {{Non-rigid Stereo-motion}},
  booktitle = {Openalex Unknown},
  year = {2007},
  doi = {10.5772/4937},
}

Loading related content...

Figure