Abstract
Portrait images often suffer from undesirable shadows cast by casual objects or even the face itself. While existing methods for portrait shadow removal require training on a large-scale synthetic dataset, we propose the first unsupervised method for portrait shadow removal without any training data. Our key idea is to leverage the generative facial priors embedded in the off-the-shelf pretrained StyleGAN2. To achieve this, we formulate the shadow removal task as a layer decomposition problem: a shadowed portrait image is constructed by the blending of a shadow image and a shadow-free image. We propose an effective progressive optimization algorithm to learn the decomposition process. Our approach can also be extended to portrait tattoo removal and watermark removal. Qualitative and quantitative experiments on a real-world portrait shadow dataset demonstrate that our approach achieves comparable performance with supervised shadow removal methods.
Introduction
Shadows in portrait photography—whether cast by external objects like hats or hands, or "self-shadows" from facial features like the nose—are notoriously difficult to remove. Existing state-of-the-art solutions typically rely on supervised learning, which requires thousands of paired images (shadowed vs. shadow-free). Collecting such data is extremely challenging, requiring perfectly controlled lighting and immobile subjects. Furthermore, models trained on synthetic data often fail to generalize to the complex shadows found in real-world scenarios.
This work addresses these challenges by proposing an unsupervised method that removes shadows using only the information within a single image and the general facial knowledge contained in a pretrained generative model.
Methodology
The researchers formulate shadow removal as an image decomposition problem. They model a shadowed image ($I$) as a blend of a shadow-free image ($I_{free}$) and a "full-shadow" image ($I_{full}$), controlled by a spatial shadow mask ($M$): $$I = I_{free} \odot M + I_{full} \odot (1 - M)$$
To solve this ill-posed problem, the framework utilizes three primary components:
- Generative Priors: A pretrained StyleGAN2 model serves as the "prior." Since it was trained on millions of clean faces (FFHQ dataset), it can generate a realistic, shadow-free face that matches the subject's identity through GAN inversion.
- MaskNet: A separate neural network that learns to generate the spatial shape and intensity of the shadow mask ($M$) from random noise.
- Color Matrix: To create the $I_{full}$ layer, a learnable $3 \times 3$ color matrix is applied to the shadow-free image, simulating light attenuation in shadowed regions.
The optimization happens in three progressive stages:
- Stage 1: Finding a latent code in StyleGAN2 that matches the input face's identity.
- Stage 2: Fixing the face and optimizing the MaskNet and Color Matrix to fit the shadow to the input image.
- Stage 3: Jointly fine-tuning all components, including specific facial features, to ensure high-fidelity reconstruction.
Extensions: Tattoo and Watermark Removal
Because the method is unsupervised and based on layer decomposition, it is highly versatile. By slightly modifying the blending equation, the authors demonstrate its effectiveness in removing other facial artifacts, such as tattoos and watermarks, without any task-specific training.
Experimental Results
The method was evaluated on a real-world portrait shadow dataset containing 100 images.
- Quantitative Performance: The unsupervised method achieved a 0.820 SSIM (Structural Similarity) and competitive LPIPS scores, outperforming general supervised shadow removal models like ST-CGAN and DHAN. This performance is comparable to the 0.859 SSIM of the leading supervised portrait-specific model (PSM), despite using no training pairs.
- Qualitative Performance: The approach successfully handles both "foreign" shadows (e.g., from a hand) and "self" shadows while maintaining the underlying skin texture and subject identity.
Limitations and Conclusion
While powerful, the method has limitations. Fine-grained details like specific wrinkles or unique facial hair can sometimes be smoothed out if they are not well-represented in the StyleGAN2 latent space. Additionally, out-of-distribution accessories (e.g., unique jewelry) may be difficult to reconstruct.
In conclusion, this work demonstrates that leveraging generative priors is a highly effective strategy for unsupervised portrait restoration. By "inverting" a GAN that already knows what clean faces look like, complex artifacts like shadows and tattoos can be removed with high fidelity.
Summary updated