Fitting 3D Morphable Models using Local Features

Patrik Huber, Zhen-Hua Feng, William J. Christmas, Josef Kittler, Matthias Rätsch

Abstract

This paper presents a novel approach for fitting 3D Morphable Models (3DMMs) to 2D images by leveraging robust local image features. Traditional gradient-based optimization is often hindered by the non-differentiable nature of feature extraction operators like SIFT. We overcome this obstacle by employing a learning-based cascaded regression framework that learns the optimization trajectory directly from data. Our method simultaneously estimates 3D shape and pose parameters in near real-time, demonstrating significant robustness to initialization and varying imaging conditions.

Introduction

Reconstructing 3D facial geometry from a single 2D image is a fundamental challenge in computer vision. While 3D Morphable Models (3DMMs) provide a powerful prior for this task, fitting them accurately and efficiently remains difficult. Existing methods typically rely on either fast linear methods using sparse 2D landmarks or slow, complex non-linear methods that use raw pixel colors and edges.

Local image features, such as SIFT or HOG, have revolutionized 2D landmark detection due to their robustness to lighting and pose variations, but they have seen limited use in 3DMM fitting. The primary roadblock is that feature extraction operators are non-differentiable, preventing the use of standard gradient-based optimization. This work introduces a cascaded regression approach to bridge this gap, enabling the use of robust local features for direct 3D model fitting.

Methodology

The proposed method utilizes a cascaded regression framework to iteratively update the 3DMM parameters. Instead of calculating analytical gradients, the system learns a series of linear regressors that map extracted local features to parameter updates.

Cascaded Regression Framework

The fitting process begins with an initial estimate of the pose and shape. In each step of the cascade, a subset of the 3D model's vertices is projected onto the 2D image plane based on the current parameters. Local SIFT features are then extracted at these projected coordinates.

Visualization of local feature extraction points projected from the 3D model onto the 2D image.

The extracted features are concatenated and fed into a regressor to predict an update for both the pose (rotation and translation) and the shape coefficients (PCA weights). This process is repeated across several stages, with each stage refining the parameters further. By jointly optimizing for shape and pose, the model achieves better alignment as the 3D geometry adapts to the specific identity in the image, which in turn improves the accuracy of subsequent feature extractions.

Experimental Results

The method was evaluated on both synthetic 3DMM data and the real-world CMU PIE database to test performance under varying illumination.

Convergence and Robustness

A key strength of the cascaded regression approach is its robustness to poor initialization. Experiments on synthetic data show that even when starting from a "worst-case" pose estimate (e.g., 11° error), the algorithm consistently converges toward the ground truth within just a few regressor steps.

Mean absolute angle error across regressor steps for uniform and worst-case initializations.

Performance and Speed

On the CMU PIE dataset, the model demonstrated strong generalization to unseen lighting conditions, achieving a mean absolute pose error of 0.86° and a shape similarity score of 0.84. Furthermore, the unoptimized implementation runs at approximately 200ms per image, making it highly viable for near real-time applications.

Discussion and Conclusion

By bypassing the non-differentiability of local features through learning, this method effectively unifies 3D model fitting with robust feature extraction. The joint optimization of shape and pose not only simplifies the pipeline but also improves accuracy, as better shape estimation leads to more precise feature localization.

Future work will focus on optimizing the runtime by exploring faster feature descriptors or dimensionality reduction techniques. This framework provides a scalable and robust foundation for real-time 3D face analysis from monocular images, with the cascaded regression library available as an open-source resource.

Summary updated

Abstract

In this paper, we propose a novel fitting method that uses local image features to fit a 3D Morphable Model to 2D images. To overcome the obstacle of optimising a cost function that contains a non-differentiable feature extraction operator, we use a learning-based cascaded regression method that learns the gradient direction from data. The method allows to simultaneously solve for shape and pose parameters. Our method is thoroughly evaluated on Morphable Model generated data and first results on real data are presented. Compared to traditional fitting methods, which use simple raw features like pixel colour or edge maps, local features have been shown to be much more robust against variations in imaging conditions. Our approach is unique in that we are the first to use local features to fit a Morphable Model. Because of the speed of our method, it is applicable for realtime applications. Our cascaded regression framework is available as an open source library (https://github.com/patrikhuber).
BibTeX
@inproceedings{huber2015fitting,
  author = {Patrik Huber and Zhen-Hua Feng and William J. Christmas and Josef Kittler and Matthias Rätsch},
  title = {{Fitting 3D Morphable Models using Local Features}},
  booktitle = {ICIP},
  year = {2015},
  url = {https://openreview.net/forum?id=61N8tZjE_47},
}
Figure