Taking a great portrait outdoors is a gamble. Even with a high-end smartphone, you’re often at the mercy of harsh sunlight that casts distracting "foreign" shadows from nearby trees or hats, and unflattering "facial" shadows—like that dark triangle under the nose. While professional photographers use scrims and reflectors to tame these shadows, casual shooters are usually stuck with what they get.
In their paper "Portrait Shadow Manipulation," Xuaner Zhang and her colleagues from UC Berkeley, Google Research, and MIT present a clever two-stage neural framework that gives casual photographers professional-grade control over lighting after the photo is taken.
TL;DR
The researchers developed a pair of neural networks that can automatically remove distracting external shadows and soften harsh facial self-shadows in portraits. By combining a sophisticated synthetic data pipeline with an explicit "facial symmetry" cue, their model transforms poorly lit "in-the-wild" photos into aesthetically pleasing portraits that look like they were shot in a studio.
What Problem Does This Paper Address?
The core challenge is that not all shadows are created equal. The authors categorize portrait shadows into two types:
- Foreign Shadows: Cast by external objects (trees, fences, hands). These are "unconstrained"—they can be any shape and are usually purely distracting.
- Facial Shadows: Cast by the face onto itself (nose shadows, eye socket shadows). These are "constrained" by facial geometry and provide important cues about the face's 3D shape, but they can be too harsh or dark.
Existing general-purpose shadow removal tools often fail on faces because they don't understand human skin's unique properties (like subsurface scattering) or the underlying geometry of a face. Conversely, "portrait relighting" methods often struggle when the input image already contains complex, high-contrast shadows.
Methods
The authors propose a modular pipeline using two separate GridNet architectures. Both networks output a per-pixel affine transformation (a scaling factor $A$ and an offset $B$) to modify the original image, ensuring that the underlying texture and details are preserved.
1. Foreign Shadow Removal
This network is trained to identify and completely erase external shadows. Because real-world "ground truth" (the same photo with and without a shadow) is nearly impossible to capture at scale, the team built a massive synthetic dataset. They didn't just overlay black shapes; they modeled subsurface scattering (the way light bleeds through skin, creating red fringes at shadow edges) and spatial variation in shadow intensity.
2. Facial Shadow Softening & Fill Light
Instead of removing facial shadows, this network "softens" them, emulating a photographer’s scrim. It also adds a virtual "fill light" to brighten the dark side of the face.
- The Symmetry Trick: To help the network "guess" what a shadowed area should look like, the authors provide a "mirrored" version of the face as an additional input. They use facial landmarks and a "soft" warping technique (Radial Basis Functions) to align the lit side of the face with the shadowed side, allowing the network to "borrow" pixels from the better-lit half.
- User Control: The model includes two "knobs"—parameters for light size (softness) and fill light intensity—allowing for adjustable results.
Datasets
- Foreign-syn: 500,000 synthetic images created by projecting 300+ real-world object silhouettes onto 5,000 "in-the-wild" face images.
- Facial-syn: Generated using Light Stage (OLAT) data. This involves high-fidelity scans of 85 subjects lit from 304 different directions, allowing the researchers to mathematically "render" perfect ground-truth pairs of harsh vs. soft lighting.
- Foreign-real: A new evaluation set captured via video. By filming a stationary subject while moving an object to cast shadows, they obtained perfectly aligned "shadowed" and "shadow-free" frames for testing.
Key Insights and Ideas
- Shadow Taxonomy Matters: Treating foreign and facial shadows as different problems is the "secret sauce." Foreign shadows are noise to be removed; facial shadows are signals to be refined.
- Symmetry as a Prior: Faces are remarkably symmetric in reflectance and geometry, but lighting rarely is. Using a warped mirror image as a neural input is a powerful way to provide the network with a "reference" for what skin should look like under the shadow.
- The "Red Fringe" of Realism: The authors realized that synthetic shadows look "fake" to a network unless you model the wavelength-dependent scattering of human skin. Including this in the training data was vital for the model to generalize to real photos.
Key Results
The model significantly outperforms state-of-the-art general shadow removal and relighting methods:
- Quantitative: On the
foreign-syndataset, their model achieved a PSNR of 29.81, compared to 22.40 for the next best deep learning competitor (Cun et al. 2020). - Qualitative: The results (see Fig. 1 and 7) show a remarkable ability to erase complex shadows (like those from a fence) while keeping the skin tone natural and the facial features sharp.
- Relighting Pre-processor: The authors showed that their tool can be used as a "clean-up" step to improve the performance of other portrait relighting algorithms.
Limitations and Future Work
- Fine Details: The model sometimes struggles with extremely fine-grained shadows, such as those cast by individual strands of hair.
- Ambiguity: In some cases, the model might confuse a very harsh facial shadow for a foreign one and try to remove it entirely rather than softening it.
- Dataset Diversity: While the Light Stage data is high-quality, it only covers 85 subjects. Expanding this to a more diverse range of ethnicities and ages would likely improve generalization further.
This work marks a significant step toward "studio-in-your-pocket" software, moving beyond simple filters to true computational lighting control.
Summary updated