Abstract
Shadow removal is an important computer vision task aiming at the detection and successful removal of the shadow produced by an occluded light source and a photo-realistic restoration of the image contents. Decades of research produced a multitude of hand-crafted restoration techniques and, more recently, learned solutions from shadowed and shadow-free training image pairs. In this work, we propose a single image shadow removal solution via self-supervised learning by using a conditioned mask. We rely on self-supervision and jointly learn deep models to remove and add shadows to images. We derive two variants for learning from paired images and unpaired images, respectively. Our validation on the recently introduced ISTD and USR datasets demonstrate large quantitative and qualitative improvements over the state-of-the-art for both paired and unpaired learning settings.
Introduction
Shadows present a significant hurdle for downstream computer vision tasks such as object tracking and semantic segmentation. They introduce complex degradations in illumination, color, and noise that are difficult to model with hand-crafted techniques. While deep learning has shown promise, the scarcity of perfectly aligned paired datasets (scenes with and without shadows) limits the scalability of supervised methods. Furthermore, existing generative models often introduce artifacts like blurriness or checkerboard patterns during restoration.
This paper addresses these challenges by introducing a robust self-supervised framework that leverages both paired and unpaired data. By framing shadow removal and addition as inverse operations within a CycleGAN-inspired architecture, the proposed method achieves high-fidelity restorations that preserve underlying textures and semantic structures.
Methodology
The proposed system utilizes two generators: $G_f$ for shadow removal and $G_s$ for shadow addition. The framework is built on the principle of bijectivity, assuming that a shadow-free image can be transformed into a shadowed one and back again without loss of information.
Key Innovations
- Conditioned Masking: Unlike standard domain translation, $G_s$ is conditioned on a binary mask. This turns shadow addition into a controlled process, which is particularly useful in unpaired settings where masks are sampled from a buffer of known shadow shapes.
- Self-Supervised Shadow ($S^3$) Loss: The model enforces mask consistency. If a shadow is added using a specific mask, the model must be able to recover that mask by comparing the generated shadowed image with the original shadow-free input.
- Perceptual Ensemble Loss: To avoid the blurriness associated with standard pixel-wise losses (L1/L2), the authors employ a combination of color loss (Gaussian-blurred MSE) and content/style losses derived from VGG-16 features. This ensures the model prioritizes visual realism and structural integrity over exact pixel matching, which is often impossible due to slight misalignments in paired datasets.
Datasets and Evaluation
The model was validated on three major benchmarks:
- ISTD: A standard dataset of 1,870 image triplets (shadowed, shadow-free, and mask).
- ISTD+: A refined version of ISTD that corrects for illumination inconsistencies in the ground truth.
- USR: A dataset designed for unpaired learning, containing separate collections of shadowed and shadow-free images from different scenes.
Results and Discussion
The proposed method significantly outperformed previous state-of-the-art (SOTA) models in both paired and unpaired settings. In unpaired learning on the ISTD dataset, the model achieved an RMSE of 7.58, a substantial improvement over CycleGAN (25.4) and MaskShadowGAN (28.34). Even in paired settings, it led with an RMSE of 15.05 compared to 22.11 for ST-CGAN.
Qualitative analysis reveals that the model effectively restores underlying textures without the "ghosting" artifacts common in previous GAN-based methods. The use of perceptual losses allows the model to handle complex illumination and semantic variations more gracefully than models relying solely on L1/L2 losses. The lower LPIPS scores further confirm that the generated images are more visually similar to ground truth from a human perspective.
Conclusion
By combining conditioned masking with a bijective cycle-consistency framework, this work provides a powerful solution for single-image shadow removal. The ability to learn effectively from unpaired data significantly reduces the burden of data collection. While the model occasionally struggles with extremely sharp color variations in limited datasets, its overall performance sets a new benchmark for the field. Future research will explore increasing mask diversity to further refine the model's control over complex shadow geometries.
Summary updated