Deep Single-Image Portrait Relighting

Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, David JacobsICCV 2019 · OpenReview

Abstract: Conventional physically-based methods for relighting portrait images need to solve an inverse rendering problem, estimating face geometry, reflectance and lighting. However, the inaccurate estimation of face components can cause strong artifacts in relighting, leading to unsatisfactory results. In this work, we apply a physically-based portrait relighting method to generate a large scale, high quality, “in the wild” portrait relighting dataset (DPR). A deep Convolutional Neural Network (CNN) is then trained using this dataset to generate a relit portrait image by using a source image and a target lighting as input. The training procedure regularizes the generated results, removing the artifacts caused by physically-based relighting methods. A GAN loss is further applied to improve the quality of the relit portrait image. Our trained network can relight portrait images with resolutions as high as 1024 × 1024. We evaluate the proposed method on the proposed DPR datset, Flickr portrait dataset and Multi-PIE dataset both qualitatively and quantitatively. Our experiments demonstrate that the proposed method achieves state-of-the-art results.

1. Introduction

Portrait relighting—the process of altering the illumination of a subject's face post-capture—remains a fundamental challenge in computer vision. Traditional methods typically rely on "inverse rendering," which involves explicitly estimating 3D geometry, skin reflectance (albedo), and the original lighting environment. However, minor inaccuracies in these estimations often lead to significant visual artifacts, such as "ghosting" or misaligned shadows. Furthermore, existing deep learning solutions frequently struggle with high-resolution outputs, often limited to low resolutions like 128x128 pixels.

2. Methodology

This work introduces a hybrid framework that integrates physical rendering principles with generative deep learning to achieve high-resolution (1024x1024) relighting from a single image.

2.1 Data Generation and Alignment

To train the model, the authors developed the Deep Portrait Relighting (DPR) dataset, containing over 138,000 images. They utilized a physically-based "Ratio Image" (RI) algorithm on the CelebA-HQ dataset. A critical innovation is the use of As-Rigid-As-Possible (ARAP) warping to align estimated 3D face normals with 2D image features, effectively eliminating the "halo" artifacts common in standard 3D face estimation.

2.2 Network Architecture and Training

The core architecture is an Hourglass Network that accepts a source image and target lighting (represented via Spherical Harmonics). To ensure the network learns a robust disentangled representation of identity and lighting, the authors employed a "skip training" strategy. By initially withholding skip connections and adding them incrementally during training, the bottleneck layer is forced to learn meaningful facial features before fine details are reintroduced.

2.3 Adversarial Refinement

To bridge the gap between synthetic training data and real-world photography, a GAN (Generative Adversarial Network) loss is applied using the FFHQ dataset. This allows the network to "learn to be better than its teacher," refining the output to remove artifacts inherent in the physical RI model and producing more realistic skin textures and shadows.

3. Experimental Results

The proposed method was evaluated on the DPR, Multi-PIE, and Flickr portrait datasets, demonstrating superior performance in both controlled and "in-the-wild" scenarios.

Relighting results under various target lighting environments. The top row shows the input and relit images; the bottom row shows the corresponding target lighting spheres.

3.1 Quantitative and Qualitative Performance

On the Multi-PIE dataset, the model achieved a Scale-invariant Mean Squared Error (Si-MSE) of 0.00590, significantly outperforming previous benchmarks like SfSNet (0.00659). Qualitatively, the method produces sharp "attached shadows" (e.g., shadows cast by the nose or brow) that were previously unattainable with deep learning methods. It is the first deep learning approach to successfully demonstrate high-fidelity relighting at 1024x1024 resolution.

4. Conclusion and Future Work

While the method achieves state-of-the-art results, limitations remain in handling extreme profile poses and complex regions such as hair, teeth, and eyes, which deviate from the assumed Lambertian reflectance model. Future research will focus on more sophisticated reflectance modeling to further enhance realism in these challenging areas.

Summary updated

Abstract

Conventional physically-based methods for relighting portrait images need to solve an inverse rendering problem, estimating face geometry, reflectance and lighting. However, the inaccurate estimation of face components can cause strong artifacts in relighting, leading to unsatisfactory results. In this work, we apply a physically-based portrait relighting method to generate a large scale, high quality, “in the wild” portrait relighting dataset (DPR). A deep Convolutional Neural Network (CNN) is then trained using this dataset to generate a relit portrait image by using a source image and a target lighting as input. The training procedure regularizes the generated results, removing the artifacts caused by physically-based relighting methods. A GAN loss is further applied to improve the quality of the relit portrait image. Our trained network can relight portrait images with resolutions as high as 1024 × 1024. We evaluate the proposed method on the proposed DPR datset, Flickr portrait dataset and Multi-PIE dataset both qualitatively and quantitatively. Our experiments demonstrate that the proposed method achieves state-of-the-art results. Please refer to https://zhhoper.github.io/dpr.html for dataset and code.
BibTeX
@inproceedings{zhou2019deep,
  author = {Hao Zhou and Sunil Hadap and Kalyan Sunkavalli and David Jacobs},
  title = {{Deep Single-Image Portrait Relighting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=2Rf1BckC3Hz},
}
Figure