Internet Based Morphable Model

Ira Kemelmacher-ShlizermanICCV 2013 · OpenReview

Abstract

In this paper, we present a new concept of building a morphable model directly from photos on the Internet. Morphable models have shown very impressive results more than a decade ago and could potentially have a huge impact on all aspects of face modeling and recognition. One of the challenges, however, is to capture and register 3D laser scans of a large number of people and facial expressions. Nowadays, there are enormous amounts of face photos on the Internet, a large portion of which has semantic labels. We propose a framework to build a morphable model directly from photos; the framework includes dense registration of Internet photos, as well as new single-view shape reconstruction and modification algorithms.

Introduction

Since their introduction in 1999, 3D Morphable Models (3DMMs) have served as the gold standard for recovering 3D face geometry from 2D images. However, the development of these models has long been hindered by a significant "data bottleneck." Traditional 3DMMs rely on high-fidelity 3D laser scans, which are expensive to acquire and typically limited to a small, non-diverse demographic (often around 200 young adults). This reliance limits the range of captured expressions, ages, and ethnicities, while the manual alignment of these scans remains a painstaking process.

This research explores a more scalable alternative: leveraging the billions of unconstrained photos available on the Internet to build a comprehensive 3D model without the need for controlled laboratory scans.

Methodology

The proposed framework replaces the traditional 3D scanning phase with an automated pipeline that processes "in-the-wild" imagery.

Overview of the Internet-based Morphable Face Model pipeline, from data collection to 3D output modification.

Data Collection and Alignment

The process begins by scraping thousands of images using semantic search terms such as "smiling baby" or "puckered lips." These semantic labels act as a proxy for 3D deformation. To establish correspondence across these diverse images, the authors utilize Collection Flow (CF). CF enables dense, pixel-wise alignment, allowing the system to compare photos across different identities and lighting conditions by mapping them to a common reference frame.

Double SVD Factorization

The technical core of the framework is a novel double SVD factorization algorithm. By assuming a Lambertian reflectance model, the authors demonstrate that a matrix of aligned pixel intensities can be modeled as a rank-$4K$ matrix. This iterative algorithm decomposes the intensity matrix into $K$ 3D basis shapes, lighting coefficients, and deformation coefficients. This approach effectively separates lighting from geometry, solving a classic ambiguity in computer vision.

Results and Key Insights

The framework was evaluated quantitatively using the YaleB dataset, which provides ground-truth 3D shapes via photometric stereo. The Internet-based model achieved a reconstruction error of approximately 4%, significantly outperforming previous single-view state-of-the-art methods that typically yielded errors in the 6–7% range.

The "Baby" Test Case

A key insight of the study is the use of infants as a primary test case. Babies are notoriously difficult to capture with 3D laser scanners because they do not remain still, yet they are ubiquitous in online photo collections.

Example of a 'laughing baby' image from the Internet dataset, representing a challenging case for traditional 3D scanning.

Qualitatively, the model successfully reconstructed highly non-rigid expressions—such as screaming and laughing—across various ethnicities. Traditional 3DMMs often fail in these cases, producing overly "average" results. Furthermore, the framework enables seamless expression synthesis, allowing users to modify the 3D expression of a 2D input while preserving the subject's identity and original lighting.

Conclusion

By treating the Internet as a massive 3D sensor, this framework overcomes the scalability issues of traditional laser-scan-based modeling. While the current model assumes Lambertian surfaces and relies on accurate semantic labels, it demonstrates a clear path toward capturing the full diversity of human facial variation. As the volume of labeled data on the Internet continues to grow, this approach offers a more scalable and automated future for 3D face modeling and recognition.

Summary updated

Abstract

In this paper we present a new concept of building a morphable model directly from photos on the Internet. Morphable models have shown very impressive results more than a decade ago, and could potentially have a huge impact on all aspects of face modeling and recognition. One of the challenges, however, is to capture and register 3D laser scans of large number of people and facial expressions. Nowadays, there are enormous amounts of face photos on the Internet, large portion of which has semantic labels. We propose a framework to build a morph able model directly from photos, the framework includes dense registration of Internet photos, as well as, new single view shape reconstruction and modification algorithms.
BibTeX
@inproceedings{kemelmachershlizerman2013internet,
  author = {Ira Kemelmacher-Shlizerman},
  title = {{Internet Based Morphable Model}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  url = {https://openreview.net/forum?id=SJWVzZG_ZH},
}
Figure