Portrait Shadow Manipulation

Xuaner Cecilia Zhang, Jonathan T. Barron, Yun-Ta Tsai, Rohit Pandey, Xiuming Zhang, Ren Ng, David E. Jacobs

Abstract

Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. Aesthetic qualities such as the position and softness of shadows and the lighting ratio between the bright and dark parts of the face are frequently determined by the constraints of the environment rather than by the photographer. Professionals address this issue by adding light shaping tools such as scrims, bounce cards, and flashes. In this paper, we present a computational approach that gives casual photographers some of this control, thereby allowing poorly-lit portraits to be relit post-capture in a realistic and easily-controllable way. Our approach relies on a pair of neural networks—one to remove foreign shadows cast by external objects, and another to soften facial shadows cast by the features of the subject and to add a synthetic fill light to improve the lighting ratio. To train our first network we construct a dataset of real-world portraits wherein synthetic foreign shadows are rendered onto the face, and we show that our network learns to remove those unwanted shadows. To train our second network we use a dataset of Light Stage scans of human subjects to construct input/output pairs of input images harshly lit by a small light source, and variably softened and fill-lit output images of each face. We propose a way to explicitly encode facial symmetry and show that our dataset and training procedure enable the model to generalize to images taken in the wild. Together, these networks enable the realistic and aesthetically pleasing enhancement of shadows and lights in real-world portrait images.

Introduction

Capturing a high-quality portrait in outdoor settings is often a challenge due to harsh, uncontrollable lighting. Distracting "foreign" shadows from trees, fences, or accessories like hats can obscure facial features, while "facial" self-shadows (e.g., under the nose or in eye sockets) can appear unflatteringly dark. While professional photographers mitigate these issues using physical modifiers like scrims and reflectors, casual photographers lack these tools.

A portrait obscured by complex "foreign" shadows cast by foliage.

This research addresses the problem by categorizing shadows into two distinct types:

  1. Foreign Shadows: Unconstrained shadows cast by external objects (trees, fences, hands).
  2. Facial Shadows: Geometrically constrained shadows cast by the face onto itself (nose shadows, eye socket shadows).

Existing shadow removal tools often fail on human subjects because they do not account for the unique optical properties of skin, such as subsurface scattering, or the underlying 3D geometry of the face.

Methodology

The proposed framework utilizes a modular pipeline consisting of two separate neural networks based on the GridNet architecture. Both networks are designed to output per-pixel affine transformations (scaling $A$ and offset $B$), which are applied to the original image to modify lighting while preserving high-frequency textures and details.

Foreign Shadow Removal

The first network identifies and eliminates external shadows. Training this model required a massive synthetic dataset (foreign-syn) of 500,000 images. To ensure realism, the researchers modeled subsurface scattering—the red fringes often seen at the edges of shadows on skin—and spatial variations in shadow intensity.

Facial Shadow Softening and Fill Light

The second network focuses on "softening" facial shadows to emulate the effect of a photographer’s scrim and adding a virtual "fill light" to balance the lighting ratio.

  • Facial Symmetry Prior: To reconstruct shadowed areas, the model uses a warped "mirrored" version of the face as input. By aligning the lit side of the face with the shadowed side using facial landmarks and Radial Basis Functions, the network can "borrow" pixel information from the better-lit half.
  • User Control: The system provides adjustable parameters for light source size (softness) and fill light intensity.

Datasets

The models were trained and evaluated using three primary datasets:

  • Foreign-syn: 500,000 synthetic images created by projecting real-world silhouettes onto 5,000 in-the-wild faces.
  • Facial-syn: High-fidelity Light Stage (OLAT) data of 85 subjects, providing ground-truth pairs for harsh vs. soft lighting.
  • Foreign-real: A video-based evaluation set with perfectly aligned shadowed and shadow-free frames.

Results

The two-stage approach significantly outperforms general-purpose shadow removal and relighting methods. Quantitatively, the model achieved a PSNR of 29.81 on the foreign-syn dataset, a substantial improvement over previous deep learning competitors.

Comparison of input portraits with the enhanced outputs produced by the proposed two-stage neural framework.

Qualitatively, the framework demonstrates a robust ability to erase complex shadows (e.g., from fences or hats) while maintaining natural skin tones and sharp facial details. Furthermore, the system can serve as a pre-processor to improve the performance of other portrait relighting algorithms.

Conclusion

By treating foreign and facial shadows as distinct problems and leveraging facial symmetry as a geometric prior, this work provides a powerful tool for post-capture lighting control. While challenges remain in handling extremely fine details like individual hair strands, this framework represents a significant step toward achieving studio-quality lighting in casual mobile photography.

Summary updated

Abstract

Casually-taken portrait photographs often suffer from unflattering lighting and shadowing because of suboptimal conditions in the environment. Aesthetic qualities such as the position and softness of shadows and the lighting ratio between the bright and dark parts of the face are frequently determined by the constraints of the environment rather than by the photographer. Professionals address this issue by adding light shaping tools such as scrims, bounce cards, and flashes. In this paper, we present a computational approach that gives casual photographers some of this control, thereby allowing poorly-lit portraits to be relit post-capture in a realistic and easily-controllable way. Our approach relies on a pair of neural networks---one to remove foreign shadows cast by external objects, and another to soften facial shadows cast by the features of the subject and to add a synthetic fill light to improve the lighting ratio. To train our first network we construct a dataset of real-world portraits wherein synthetic foreign shadows are rendered onto the face, and we show that our network learns to remove those unwanted shadows. To train our second network we use a dataset of Light Stage scans of human subjects to construct input/output pairs of input images harshly lit by a small light source, and variably softened and fill-lit output images of each face. We propose a way to explicitly encode facial symmetry and show that our dataset and training procedure enable the model to generalize to images taken in the wild. Together, these networks enable the realistic and aesthetically pleasing enhancement of shadows and lights in real-world portrait images
BibTeX
@article{zhang2020portrait,
  author = {Xuaner Cecilia Zhang and Jonathan T. Barron and Yun-Ta Tsai and Rohit Pandey and Xiuming Zhang and Ren Ng and David E. Jacobs},
  title = {{Portrait Shadow Manipulation}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  url = {https://openreview.net/forum?id=mr55Xe7BopS},
}
Figure