TL;DR
This paper introduces the first unsupervised method for removing shadows from portrait images. By leveraging the "facial priors" already embedded in a pretrained StyleGAN2 model, the researchers can decompose a single shadowed image into a clean face, a shadow mask, and a shadow-color layer—all without requiring a massive dataset of paired "shadow vs. no-shadow" training images.
What Problem Does This Paper Address?
Shadows in portrait photography—whether cast by a hat, a hand, or the nose—are notoriously difficult to remove. Existing state-of-the-art solutions typically rely on supervised learning, which requires thousands of pairs of images (one with a shadow and an identical one without).
Collecting this data is a nightmare: you need perfectly controlled lighting and subjects who don't move a muscle between shots. Furthermore, models trained on synthetic data often fail to generalize to the messy, complex shadows found in the real world. The authors tackle this by asking: Can we remove shadows using only the information contained within a single image and a general understanding of what human faces look like?
Methods
The authors formulate shadow removal as an image decomposition problem. They model a shadowed image ($I$) as a blend of a shadow-free image ($I_{free}$) and a "full-shadow" image ($I_{full}$), controlled by a shadow mask ($M$): $$I = I_{free} \odot M + I_{full} \odot (1 - M)$$
To solve this "ill-posed" problem (where many combinations of $I_{free}$ and $M$ could technically work), they use three key components:
- Generative Priors: They use a pretrained StyleGAN2 model. Because StyleGAN2 was trained on millions of clean faces, it acts as a "prior"—it knows how to generate a realistic, shadow-free face that matches the subject's identity.
- MaskNet: A small, separate neural network that learns to generate the spatial shape and intensity of the shadow mask ($M$) from random noise.
- Color Matrix: To create the $I_{full}$ layer, they apply a learnable $3 \times 3$ color matrix to the shadow-free image, simulating how light is attenuated in shadowed regions.
The optimization happens in three progressive stages:
- Stage 1: Find a latent code in StyleGAN2 that roughly matches the input face.
- Stage 2: Fix the face and optimize the MaskNet and Color Matrix to "fit" the shadow to the input image.
- Stage 3: Fine-tune everything together, including specific facial features (eyes, mouth, nose) to ensure high-fidelity details.
Datasets
- Real-world Portrait Shadow Dataset: Used for evaluation, containing 100 images of 9 subjects in various poses and lighting conditions (originally proposed by Zhang et al., 2020).
- CelebA-HQ: Used to synthesize data for the "extension" tasks (tattoo and watermark removal).
- FFHQ: The pretrained StyleGAN2 model used as the "prior" was originally trained on this dataset.
Key Insights and Ideas
- The GAN as a Teacher: The core "aha" moment is that we don't need to teach a model to remove shadows if we already have a model that knows how to generate clean faces. By "inverting" the GAN, we can find the clean version of the shadowed person.
- Progressive Optimization is Key: If you try to optimize the face, the mask, and the color all at once, the model gets confused (layer ambiguity). By starting with the face and then "adding" the shadow logic, the decomposition becomes stable.
- Versatility: Because the method is unsupervised and based on decomposition, it isn't just for shadows. By slightly modifying the blending equation, the authors show it can also remove facial tattoos and watermarks without any specific training for those tasks.
Key Results
- Quantitative: On the SSIM (Structural Similarity) and LPIPS (Perceptual Loss) metrics, this unsupervised method outperformed general supervised shadow removal models (like ST-CGAN and DHAN). It achieved a 0.820 SSIM, which is comparable to the 0.859 SSIM of the leading supervised portrait-specific model (PSM), despite having seen zero training pairs.
- Qualitative: The method successfully handles complex "foreign" shadows (e.g., from a hand) and "self" shadows (e.g., from the nose) while maintaining the underlying skin texture and identity.
Limitations and Future Work
- Detail Smoothing: Because the method relies on GAN inversion, fine-grained details like specific wrinkles or bushy beards can sometimes be "smoothed out" if the StyleGAN2 model doesn't represent them perfectly in its latent space.
- Out-of-Distribution Items: If a subject is wearing an accessory (like a unique hat or jewelry) that wasn't well-represented in the StyleGAN2 training data (FFHQ), the model may struggle to reconstruct those parts of the image correctly.
- Future Direction: The authors suggest that using more powerful GAN inversion techniques or models with higher expression ability could further improve the restoration of fine facial details.
Summary updated