Boundary-Aware Divide and Conquer: A Diffusion-based Solution for Unsupervised Shadow Removal

Lanqing Guo, Chong Wang, Wenhan Yang, Yufei Wang, Bihan Wen

Abstract

Shadow removal remains a significant challenge in computer vision, primarily due to the difficulty of obtaining paired datasets for supervised learning. This paper presents Boundary-aware Conditional Diffusion (BCDiff), a novel unsupervised framework that leverages the generative power of pre-trained diffusion models. By adopting a "divide and conquer" strategy that separately treats shadow, non-shadow, and complex boundary regions, the proposed method achieves state-of-the-art unsupervised performance, rivaling many fully supervised alternatives.

Introduction

Deep learning has significantly advanced shadow removal, yet most high-performing models rely on supervised learning with paired "shadow/shadow-free" image datasets. Such datasets are laborious to collect and often fail to generalize to diverse real-world conditions. Unsupervised methods, while more flexible, frequently suffer from "hallucinations"—the creation of artifacts or incorrect textures—due to a lack of strong natural image priors.

Qualitative comparison of shadow removal methods. Our unsupervised approach (f) demonstrates superior boundary handling and color consistency compared to existing unsupervised methods (d, e) and is competitive with supervised results (b).

A key challenge in shadow removal is the non-uniform nature of shadows, which consist of a dark "umbra" (the center) and a blurry "penumbra" (the boundary). Standard models often struggle to address both simultaneously. BCDiff addresses this by treating shadow removal as a localized restoration problem guided by global image priors and a specific focus on boundary transitions.

Methodology: Boundary-aware Conditional Diffusion (BCDiff)

The BCDiff framework is built upon three core pillars designed to provide structural and contextual guidance during the restoration process.

1. Pre-trained Diffusion Prior

Instead of training a model from scratch, the method utilizes a pre-trained unconditional diffusion model. This model acts as a powerful "natural image prior," ensuring that the generated shadow-free regions remain visually coherent and realistic.

2. Context Conditioned Generation (CCG)

During the iterative denoising process, the model uses non-shadow regions as a context. It effectively "inpaints" the shadow areas by referencing the surrounding textures and lighting conditions. This ensures that the restored area doesn't just look like a smooth patch but has the correct underlying detail.

3. Shadow-Invariant Intrinsic Decomposition (SIID)

This "Divide and Conquer" module decomposes the image into reflectance (material property) and illumination. While illumination varies across a shadow boundary, reflectance should remain constant. By enforcing reflectance consistency, the model can recover fine structural details in dark regions that might otherwise be lost. This solves the "isolation" problem, where the center of a large shadow lacks the immediate context needed for accurate restoration.

The BCDiff architecture. The process involves iterative refinement using Context Conditioned Generation (CCG) and an Illumination-consistency constraint (IRM) to ensure the restored regions match the global scene.

Experiments and Results

The researchers evaluated BCDiff on the ISTD Dataset and a Video Shadow Removal Dataset. Despite being unsupervised and training-free for the removal task itself, the method demonstrated remarkable performance.

Quantitative Performance

On the ISTD benchmark, BCDiff achieved a PSNR of 35.91 in shadow regions, a significant improvement over the previous unsupervised state-of-the-art, G2R (33.58 PSNR). Remarkably, it also outperformed several supervised models in terms of Root Mean Square Error (RMSE) and Structural Similarity Index (SSIM).

Qualitative Analysis

Visually, the method excels at producing clean boundaries. While competing unsupervised models often leave "ghost" edges or introduce color tints, BCDiff's results are nearly indistinguishable from ground truth. The model effectively handles the mathematical complexity of the penumbra while applying linear shifts to the simpler umbra regions.

Conclusion

BCDiff represents a significant step forward for unsupervised shadow removal. By combining the generative power of diffusion models with a boundary-aware "divide and conquer" strategy, it overcomes the common pitfalls of GAN-based unsupervised methods. While the method currently faces limitations regarding computational speed (requiring 250 timesteps) and a dependency on shadow masks, its ability to match supervised performance without paired training data marks a major milestone in the field.

Summary updated

Abstract

Recent deep learning methods have achieved superior results in shadow removal. However, most of these supervised methods rely on training over a huge amount of shadow and shadow-free image pairs, which require laborious annotations and may end up with poor model generalization. Shadows, in fact, only form partial degradation in images, while their non-shadow regions provide rich structural information potentially for unsupervised learning. In this paper, we propose a novel diffusion-based solution for unsupervised shadow removal, which separately modeling the shadow, non-shadow, and their boundary regions. We employ a pretrained unconditional diffusion model fused with non-corrupted information to generate the natural shadow-free image. While the diffusion model can restore the clear structure in the boundary region by utilizing its adjacent non-corrupted contextual information, it fails to address the inner shadow area due to the isolation of the non-corrupted contexts. Thus we further propose a Shadow-Invariant Intrinsic Decomposition module to exploit the underlying reflectance in the shadow region to maintain structural consistency during the diffusive sampling. Extensive experiments on the publicly available shadow removal datasets show that the proposed method achieves a significant improvement compared to existing unsupervised methods, and even is comparable with some existing supervised methods.
BibTeX
@inproceedings{guo2023boundary,
  author = {Lanqing Guo and Chong Wang and Wenhan Yang and Yufei Wang and Bihan Wen},
  title = {{Boundary-Aware Divide and Conquer: A Diffusion-based Solution for Unsupervised Shadow Removal}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01199},
}
Figure