Abstract
In this paper, we address the problem of simultaneously recovering the 3D shape and pose of a deformable and potentially elastic object from 2D motion. This is a highly ambiguous problem typically tackled by using low-rank shape and trajectory constraints. We show that formulating the problem in terms of a low-rank force space that induces the deformation, allows for a better physical interpretation of the resulting priors and a more accurate representation of the actual object's behavior. However, this comes at the price of, besides force and pose, having to estimate the elastic model of the object. For this, we use an Expectation Maximization strategy, where each of these parameters are successively learned within partial M-steps, while robustly dealing with missing observations. We thoroughly validate the approach on both mocap and real sequences, showing more accurate 3D reconstructions than state-of-the-art, and additionally providing an estimate of the full elastic model with no a priori information.
Introduction
The task of Non-Rigid Structure from Motion (NRSfM) involves reconstructing the 3D shape and camera pose of a deforming object from a monocular 2D video sequence. This problem is inherently ill-posed because many different 3D shapes can produce the same 2D projection. Traditional approaches mitigate this ambiguity by imposing low-rank constraints on shape or trajectory. While effective, these statistical priors often lack physical grounding, acting as "black boxes" that do not account for the underlying mechanics of deformation. This research proposes a shift toward physics-based modeling by introducing the concept of "Force Space."
The Force Space Formulation
The core innovation of this work is modeling 3D deformation as a result of external forces acting on an elastic body. Instead of learning abstract shape bases, the model factorizes the 3D shape $S$ into physically interpretable components: $$S = C \cdot F \cdot \Gamma$$ where $C$ is the Compliance Matrix representing the object's internal elastic properties, $F$ is a Force Basis representing a low-rank set of forces, and $\Gamma$ contains the time-varying coefficients.
This formulation provides a mathematical duality between standard statistical models and physical force-based models, effectively giving physical meaning to previously ungrounded methods. By learning the compliance matrix $C$, the model captures how different parts of an object (e.g., a face or a cloth) respond to force.
Learning and Optimization
To estimate the unknown parameters—camera pose, force basis, and the compliance matrix—the authors employ a probabilistic Expectation-Maximization (EM) framework.
- E-step: Estimates the most likely forces given the observed 2D point tracks.
- M-step: Updates the camera motion and the object's physical parameters.
A significant advantage of this global, physics-based approach is its inherent robustness to missing data. When points are occluded, the underlying elastic model "fills in" the missing geometry based on the learned physical constraints.
Experimental Results
The proposed method, EM-PFS, was evaluated against eight state-of-the-art NRSfM techniques using both motion capture (mocap) data and real-world video sequences.
Quantitative Performance
On mocap datasets including facial expressions (Jacky, Face), articulated motion (Walking), and cloth deformation (Flag), EM-PFS achieved an average 3D reconstruction error of 4.62%, outperforming the next best competitor, SPM (5.09%).
Real-World Robustness
The model was tested on challenging real-world sequences such as medical footage of a beating heart and American Sign Language (ASL) videos. In the ASL sequence, which contains over 11% missing data due to hand occlusions, the model successfully maintained a stable 3D reconstruction.
Conclusion
By integrating continuum mechanics into the NRSfM framework, this work demonstrates that physical priors can lead to more accurate and interpretable 3D reconstructions than purely statistical ones. While the current model recovers elasticity up to a scale factor, it opens the door for future research to estimate absolute material properties from monocular video. This has significant implications for fields ranging from computer vision to medical imaging and engineering.
Summary updated