TL;DR
This paper introduces a novel method for fitting 3D Morphable Face Models (3DMMs) to 2D images by leveraging local image features like SIFT. By employing a cascaded regression approach, they overcome the challenge of non-differentiable feature extraction, enabling simultaneous estimation of shape and pose parameters efficiently, even in real-time.
What Problem Does This Paper Address?
Reconstructing a 3D face from a single 2D image is a notoriously difficult problem due to the inherent ill-posedness and the many parameters involved (pose, identity, lighting, etc.). Traditional methods often rely on either simple facial landmarks or raw pixel data, which can be sensitive to variations in lighting and image conditions. While advanced local features like SIFT have proven robust for tasks like 2D landmark detection, their non-differentiable nature makes them challenging to integrate directly into optimization frameworks for 3DMM fitting. This paper aims to bridge this gap by developing a method that effectively uses these robust local features for accurate and efficient 3DMM fitting.
Methods
The core of the proposed method is a cascaded regression framework. Instead of directly differentiating a cost function involving non-differentiable local features (like SIFT), this approach learns a sequence of "weak regressors" that approximate the gradient direction from data. Each regressor takes extracted local features from the input image (around projected 3D model points) and predicts an update to the model's parameters (shape and/or pose).
- Local Feature Extraction: SIFT features are extracted from the input image at locations corresponding to projected 3D vertices of the Morphable Model.
- Cascaded Regression: A series of regressors are trained. Each regressor takes the concatenated local features and the current model parameters to predict a small update. This process is repeated iteratively, with each regressor refining the estimate from the previous one.
- Simultaneous Pose and Shape Fitting: The framework is flexible enough to simultaneously estimate both pose (rotation and translation) and shape parameters of the 3DMM. The parameter vector is extended to include shape coefficients (e.g., the first two PCA components).
This approach circumvents the need for explicit gradient calculation of the feature extraction process, making it compatible with non-differentiable features.
Datasets
- Morphable Model Generated Data: Synthetic data was generated using the 3D Morphable Model itself. This allowed for precise ground truth for shape and pose, and controlled variations in lighting and backgrounds.
- PIE Database: The authors also evaluated their method on the CMU Pose, Illumination, and Expression (PIE) database, using fitting results from a state-of-the-art method as ground truth. This allowed for evaluation on real-world images with varying illumination conditions.
Key Insights and Ideas
- Bridging the Gap: The paper successfully demonstrates that robust local features (SIFT) can be effectively used for 3DMM fitting, a task previously dominated by simpler features or landmark-based methods.
- Learning Gradients: The cascaded regression approach cleverly sidesteps the non-differentiability issue by learning the parameter updates directly from data, effectively learning the "gradient" in feature space.
- Unifying Pose and Shape: The method allows for simultaneous estimation of pose and shape, which is more efficient and potentially more accurate than iterative approaches that handle these separately.
- Potential for Real-Time: The speed of the cascaded regression framework makes the method suitable for real-time applications.
- Novelty in Unification: The authors highlight that their approach is unique in using local features for 3DMM fitting and also presents a novel step towards unifying landmark detection with 3DMM fitting.
Key Results
- Pose Fitting: Achieved accurate pose estimation, with mean absolute errors comparable to or better than traditional methods like POSIT when landmark accuracy is degraded. For synthetic data, the error was around 2°, and for PIE data, it was 0.86°.
- Simultaneous Shape and Pose Fitting: Demonstrated strong performance, achieving a shape similarity (cosine angle between coefficients) of 0.87 on synthetic data and 0.84 on the PIE database. The joint fitting slightly improved pose accuracy compared to pose-only fitting.
- Runtime: The method is fast, achieving near real-time performance (around 200 milliseconds per image with an unoptimized implementation), making it applicable for interactive applications.
Limitations and Future Work
- Limitations:
- The method relies on a rough initialisation (e.g., face box or initial landmarks), though it is robust to imprecise initializations.
- The runtime depends on the number of vertices used for feature extraction, which influences feature vector dimensionality.
- Future Work:
- Further explore how the proposed method can unify landmark detection and 3DMM fitting.
- Extend the fitting to include more shape and possibly albedo coefficients.
- Investigate faster feature extraction or compression techniques to further speed up the process.
Summary updated