Single image portrait relighting

Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E. Debevec, Ravi Ramamoorthi

TL;DR

This paper introduces a neural network that can relight portraits from a single, unconstrained cellphone image. By training on a custom light stage dataset, their model learns to realistically change the lighting of a portrait, offering a powerful tool for casual photographers and enabling new interactive applications.

What Problem Does This Paper Address?

Professional photographers meticulously control lighting to enhance portraits, but everyday users are limited by ambient conditions. While prior relighting techniques exist, they often require specialized hardware, multiple images, or accurate 3D models, making them impractical for consumer use. This paper aims to bridge this gap by enabling high-quality portrait relighting from a single, casually captured image, mimicking professional studio effects for anyone with a smartphone.

Methods

The core of the proposed system is a novel neural network architecture, dubbed the Portrait-Relighting Network (PR-Net). This network employs an encoder-decoder structure with skip connections, similar to U-Nets. Crucially, it's designed to:

  • Predict Input Illumination: The encoder not only processes the input image but also predicts its original lighting environment.
  • Inject Target Illumination: The desired target lighting environment is injected into the bottleneck of the network.
  • Generate Relit Image: The decoder then uses this information to produce the relit portrait.

The training process incorporates a weighted combination of three loss functions:

  1. Target Relighting Loss: Minimizes the difference between the generated relit image and the ground truth.
  2. Source Illumination Loss: Encourages accurate prediction of the input image's lighting.
  3. Self-Supervision Loss: A clever addition where the network is trained to reconstruct the input image using its own predicted illumination. This helps disentangle lighting from reflectance and improves overall relighting quality, even for tasks where only relighting is desired.

Datasets

The authors created a custom dataset for training and evaluation:

  • Training Data: Generated from a light stage capturing 18 individuals under 304 directional light sources. These "one-light-at-a-time" (OLAT) images were combined with 3094 natural environment illuminations (from various HDR datasets) to create approximately 226,800 paired training images.
  • Validation Data: Similar to the training data, but using 4 different individuals.
  • Real-World Evaluation: The model was also tested on hundreds of "in the wild" cellphone portraits from the Portrait-mode and HDR+ dataset.

Key Insights and Ideas

  • End-to-End Learning for Relighting: Instead of relying on explicit geometry or reflectance estimation, the network directly learns the complex mapping from input image and target lighting to the relit output. This allows it to capture subtle, non-Lambertian effects of human skin.
  • The Power of Self-Supervision: The inclusion of a self-supervision loss, where the network reconstructs the input image using its own predicted lighting, significantly improves relighting performance. This is a key insight, demonstrating that learning to "understand" the input lighting aids in generating better relit outputs.
  • Bridging the Gap Between Studio and Consumer: By training on real-world data captured in a controlled environment and then applying it to unconstrained cellphone images, the paper shows a viable path to bringing professional-level lighting control to everyday users.
  • Speed Matters: The model's ability to generate a 640x640 image in just 160ms opens the door for interactive, real-time applications on mobile devices.

Key Results

  • Quantitative Performance: The proposed model significantly outperforms prior state-of-the-art methods on their validation set for relighting accuracy, achieving up to 57-75% lower error rates compared to the next best technique.
  • Speed: The model is remarkably fast, achieving 0.16 seconds per image, which is orders of magnitude faster than some previous methods.
  • Qualitative Results: Visual comparisons show that the model produces more realistic and compelling relighting results than existing methods, even for challenging scenarios like backlighting and subsurface scattering. It also generalizes well to subjects and poses not seen in the training data.

Limitations and Future Work

  • Generalization to Extreme Cases: The model struggles with input images exhibiting very hard shadowing, sharp specularities, or highly saturated pixels, as these are underrepresented in the training data.
  • Accessories with Saturated Colors: The model may produce artifacts when presented with accessories that have saturated colors not seen during training.
  • Future Work: The authors suggest that adding more training data with hard shadows and specularities could improve robustness. Similarly, incorporating semantic or geometric understanding might help with handling accessories. They also envision their model enabling other computer vision tasks, such as generating synthetic training data for facial recognition or 3D reconstruction.

Summary updated

Abstract

Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer photographers are usually constrained to the illumination of their environment. Though prior works have explored techniques for relighting an image, their utility is usually limited due to requirements of specialized hardware, multiple images of the subject under controlled or known illuminations, or accurate models of geometry and reflectance. To this end, we present a system for portrait relighting: a neural network that takes as input a single RGB image of a portrait taken with a standard cellphone camera in an unconstrained environment, and from that image produces a relit image of that subject as though it were illuminated according to any provided environment map. Our method is trained on a small database of 18 individuals captured under different directional light sources in a controlled light stage setup consisting of a densely sampled sphere of lights. Our proposed technique produces quantitatively superior results on our dataset's validation set compared to prior works, and produces convincing qualitative relighting results on a dataset of hundreds of real-world cellphone portraits. Because our technique can produce a 640 × 640 image in only 160 milliseconds, it may enable interactive user-facing photographic applications in the future.
BibTeX
@article{sun2019single,
  author = {Tiancheng Sun and Jonathan T. Barron and Yun-Ta Tsai and Zexiang Xu and Xueming Yu and Graham Fyffe and Christoph Rhemann and Jay Busch and Paul E. Debevec and Ravi Ramamoorthi},
  title = {{Single image portrait relighting}},
  journal = {ACM Trans. Graphics},
  year = {2019},
  url = {https://openreview.net/forum?id=7xu6ZCQN8Ur},
}

Loading related content...

Figure