Portrait shadow manipulation

Xuaner Cecilia Zhang, Jonathan T. Barron, Yun-Ta Tsai, Rohit Pandey, Xiuming Zhang, Ren Ng, David E. Jacobs

Taking a great portrait outdoors is a gamble. Even with a high-end smartphone, you’re often at the mercy of harsh sunlight that casts distracting "foreign" shadows from nearby trees or hats, and unflattering "facial" shadows—like that dark triangle under the nose. While professional photographers use scrims and reflectors to tame these shadows, casual shooters are usually stuck with what they get.

In their paper "Portrait Shadow Manipulation," Xuaner Zhang and her colleagues from UC Berkeley, Google Research, and MIT present a clever two-stage neural framework that gives casual photographers professional-grade control over lighting after the photo is taken.

TL;DR

The researchers developed a pair of neural networks that can automatically remove distracting external shadows and soften harsh facial self-shadows in portraits. By combining a sophisticated synthetic data pipeline with an explicit "facial symmetry" cue, their model transforms poorly lit "in-the-wild" photos into aesthetically pleasing portraits that look like they were shot in a studio.

What Problem Does This Paper Address?

The core challenge is that not all shadows are created equal. The authors categorize portrait shadows into two types:

  1. Foreign Shadows: Cast by external objects (trees, fences, hands). These are "unconstrained"—they can be any shape and are usually purely distracting.
  2. Facial Shadows: Cast by the face onto itself (nose shadows, eye socket shadows). These are "constrained" by facial geometry and provide important cues about the face's 3D shape, but they can be too harsh or dark.

Existing general-purpose shadow removal tools often fail on faces because they don't understand human skin's unique properties (like subsurface scattering) or the underlying geometry of a face. Conversely, "portrait relighting" methods often struggle when the input image already contains complex, high-contrast shadows.

Methods

The authors propose a modular pipeline using two separate GridNet architectures. Both networks output a per-pixel affine transformation (a scaling factor $A$ and an offset $B$) to modify the original image, ensuring that the underlying texture and details are preserved.

1. Foreign Shadow Removal

This network is trained to identify and completely erase external shadows. Because real-world "ground truth" (the same photo with and without a shadow) is nearly impossible to capture at scale, the team built a massive synthetic dataset. They didn't just overlay black shapes; they modeled subsurface scattering (the way light bleeds through skin, creating red fringes at shadow edges) and spatial variation in shadow intensity.

2. Facial Shadow Softening & Fill Light

Instead of removing facial shadows, this network "softens" them, emulating a photographer’s scrim. It also adds a virtual "fill light" to brighten the dark side of the face.

  • The Symmetry Trick: To help the network "guess" what a shadowed area should look like, the authors provide a "mirrored" version of the face as an additional input. They use facial landmarks and a "soft" warping technique (Radial Basis Functions) to align the lit side of the face with the shadowed side, allowing the network to "borrow" pixels from the better-lit half.
  • User Control: The model includes two "knobs"—parameters for light size (softness) and fill light intensity—allowing for adjustable results.

Datasets

  • Foreign-syn: 500,000 synthetic images created by projecting 300+ real-world object silhouettes onto 5,000 "in-the-wild" face images.
  • Facial-syn: Generated using Light Stage (OLAT) data. This involves high-fidelity scans of 85 subjects lit from 304 different directions, allowing the researchers to mathematically "render" perfect ground-truth pairs of harsh vs. soft lighting.
  • Foreign-real: A new evaluation set captured via video. By filming a stationary subject while moving an object to cast shadows, they obtained perfectly aligned "shadowed" and "shadow-free" frames for testing.

Key Insights and Ideas

  • Shadow Taxonomy Matters: Treating foreign and facial shadows as different problems is the "secret sauce." Foreign shadows are noise to be removed; facial shadows are signals to be refined.
  • Symmetry as a Prior: Faces are remarkably symmetric in reflectance and geometry, but lighting rarely is. Using a warped mirror image as a neural input is a powerful way to provide the network with a "reference" for what skin should look like under the shadow.
  • The "Red Fringe" of Realism: The authors realized that synthetic shadows look "fake" to a network unless you model the wavelength-dependent scattering of human skin. Including this in the training data was vital for the model to generalize to real photos.

Key Results

The model significantly outperforms state-of-the-art general shadow removal and relighting methods:

  • Quantitative: On the foreign-syn dataset, their model achieved a PSNR of 29.81, compared to 22.40 for the next best deep learning competitor (Cun et al. 2020).
  • Qualitative: The results (see Fig. 1 and 7) show a remarkable ability to erase complex shadows (like those from a fence) while keeping the skin tone natural and the facial features sharp.
  • Relighting Pre-processor: The authors showed that their tool can be used as a "clean-up" step to improve the performance of other portrait relighting algorithms.

Limitations and Future Work

  • Fine Details: The model sometimes struggles with extremely fine-grained shadows, such as those cast by individual strands of hair.
  • Ambiguity: In some cases, the model might confuse a very harsh facial shadow for a foreign one and try to remove it entirely rather than softening it.
  • Dataset Diversity: While the Light Stage data is high-quality, it only covers 85 subjects. Expanding this to a more diverse range of ethnicities and ages would likely improve generalization further.

This work marks a significant step toward "studio-in-your-pocket" software, moving beyond simple filters to true computational lighting control.

Summary updated

Abstract

Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. Aesthetic qualities such as the position and softness of shadows and the lighting ratio between the bright and dark parts of the face are frequently determined by the constraints of the environment rather than by the photographer. Professionals address this issue by adding light shaping tools such as scrims, bounce cards, and flashes. In this paper, we present a computational approach that gives casual photographers some of this control, thereby allowing poorly-lit portraits to be relit post-capture in a realistic and easily-controllable way. Our approach relies on a pair of neural networks---one to remove foreign shadows cast by external objects, and another to soften facial shadows cast by the features of the subject and to add a synthetic fill light to improve the lighting ratio. To train our first network we construct a dataset of real-world portraits wherein synthetic foreign shadows are rendered onto the face, and we show that our network learns to remove those unwanted shadows. To train our second network we use a dataset of Light Stage scans of human subjects to construct input/output pairs of input images harshly lit by a small light source, and variably softened and fill-lit output images of each face. We propose a way to explicitly encode facial symmetry and show that our dataset and training procedure enable the model to generalize to images taken in the wild. Together, these networks enable the realistic and aesthetically pleasing enhancement of shadows and lights in real-world portrait images.1
BibTeX
@article{zhang2020portrait,
  author = {Xuaner Cecilia Zhang and Jonathan T. Barron and Yun-Ta Tsai and Rohit Pandey and Xiuming Zhang and Ren Ng and David E. Jacobs},
  title = {{Portrait shadow manipulation}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  url = {https://openreview.net/forum?id=mr55Xe7BopS},
}

Loading related content...

Figure