Deep Structure for end-to-end inverse rendering

Shima Kamyab, Ali Ghodsi, Zohreh AzimifararXiv 2017 · arXiv

Introduction

Inverse rendering is a fundamental challenge in computer vision, aiming to recover 3D properties such as shape and texture from 2D image inputs. In the context of human faces, traditional approaches often rely on 3D Morphable Models (3DMMs). These models represent facial geometry as a weighted combination of basis vectors typically derived via Principal Component Analysis (PCA). However, fitting these models to 2D images usually necessitates manual landmarking or computationally expensive optimization. This paper proposes a deep learning framework to automate this process, learning both the 3D shape space and the 2D-to-3D mapping concurrently.

Methodology

The proposed architecture consists of two interconnected deep networks trained simultaneously to exploit the "common cause" relationship between a 3D object and its 2D projection.

The common cause representation model, where Z is the latent space shared by 3D shapes (X3D) and 2D projections (X2D).
  1. The 3D Shape Learner (Autoencoder): Rather than relying on linear PCA, the framework employs an autoencoder (AE) to learn the latent shape space of 3D faces. The AE compresses a 3D point cloud into a low-dimensional coefficient vector and subsequently reconstructs the shape. The decoder's weights effectively serve as learned basis vectors.
  2. The 2D-to-3D Mapper (CNN): A deep Convolutional Neural Network (CNN) is trained to predict the latent coefficients directly from a single 2D image.

The networks are linked by a joint loss function ($J_1 + J_2$), where $J_1$ minimizes 3D reconstruction error and $J_2$ ensures the CNN's predictions align with the AE's latent space. This joint training acts as a regularizer, ensuring a consistent representation across both 2D and 3D domains.

Dataset and Experimental Setup

The model was trained using synthetic data generated from the Basel Face Model (BFM). The dataset comprised 10,000 synthetic 3D faces in neutral expressions. For each 3D model, a $32 \times 32$ grayscale 2D image was rendered. To enhance robustness, 20% of these images included pose variations up to 90 degrees. Each 3D face was represented as a point cloud of 53,490 vertices, resulting in a high-dimensional input vector of 160,470 coordinates.

Examples of 2D input images and their corresponding 3D point cloud representations.

Results and Discussion

The deep framework demonstrated superior performance compared to traditional landmark-based PCA methods (e.g., Aldrian et al.). Quantitatively, the proposed method achieved a Mean Squared Error (MSE) of 0.3673, a significant improvement over the baseline's 0.5248. Visual analysis via error maps confirmed that the model captures global facial structure more accurately with fewer artifacts.

Furthermore, the model exhibited stability across various poses. While MSE increased at extreme angles (approaching 90 degrees), the performance remained competitive. The use of per-vertex normal differences further validated that the orientation of the reconstructed surfaces was more precise than models trained solely on standard BFM bases.

Conclusion and Future Work

This research presents a fully automated, "one-pass" framework for 3D face reconstruction that eliminates the need for manual initialization. While successful on synthetic data, future work will focus on adapting the model for "in-the-wild" imagery, which involves greater variability in lighting and texture. Additionally, exploring adaptive latent space dimensionality—currently fixed at 280 neurons—and alternative architectures, such as Variational Autoencoders (VAEs), may further refine reconstruction fidelity.

Summary updated

Abstract

Inverse rendering in a 3D format denoted to recovering the 3D properties of a scene given 2D input image(s) and is typically done using 3D Morphable Model (3DMM) based methods from single view images. These models formulate each face as a weighted combination of some basis vectors extracted from the training data. In this paper a deep framework is proposed in which the coefficients and basis vectors are computed by training an autoencoder network and a Convolutional Neural Network (CNN) simultaneously. The idea is to find a common cause which can be mapped to both the 3D structure and corresponding 2D image using deep networks. The empirical results verify the power of deep framework in finding accurate 3D shapes of human faces from their corresponding 2D images on synthetic datasets of human faces.
BibTeX
@misc{kamyab2017deep,
  author = {Shima Kamyab and Ali Ghodsi and Zohreh Azimifar},
  title = {{Deep Structure for end-to-end inverse rendering}},
  year = {2017},
  eprint = {1708.08998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.08998},
}
Figure