SHARDS: Efficient Shadow Removal Using Dual Stage Network for High-Resolution Images

Mrinmoy Sen, Sai Pradyumna Chermala, Nazrinbanu Nurmohammad Nagori, Venkat Peddigari, Praful Mathur, B. H. Pawan Prasad, Moonhwan JeongWACV 2023

TL;DR

This paper introduces SHARDS, a novel two-stage deep learning architecture for efficient shadow removal in high-resolution images. By first processing at a lower resolution and then refining details, SHARDS achieves state-of-the-art results while being significantly more computationally efficient and deployable on mobile devices.

What Problem Does This Paper Address?

Shadows in images are a persistent challenge in computer vision, often degrading the performance of downstream tasks like object recognition and requiring tedious manual editing in professional photography. While deep learning has shown promise in shadow removal, existing methods are largely limited to low-resolution images (<1MP). This limitation stems from the computational demands and memory constraints of deep neural networks, especially when dealing with high-resolution inputs. Furthermore, many existing approaches employ heavy network architectures that are unsuitable for resource-constrained environments like smartphones. The lack of high-resolution shadow removal datasets also hinders progress in this area.

Methods

SHARDS tackles the high-resolution shadow removal problem using a two-stage approach with lightweight networks:

  • Low-resolution Shadow Removal Network (LSRNet): This network operates on a downsampled version of the input shadow image. By processing at a lower resolution, LSRNet can achieve a larger receptive field, enabling it to effectively aggregate global context from shadow-free regions to relight the shadow areas. It employs an encoder-bottleneck-decoder architecture with self-attention in the bottleneck for capturing long-range dependencies and Convolutional Block Attention (CBAM) layers in the decoder for dynamic feature weighting. Gamma correction on the input image is also used to adapt to varying shadow intensities.
  • Detail Refinement Network (DRNet): This lightweight network takes the low-resolution shadow-free output from LSRNet and the original high-resolution shadow image as guidance. Its task is to restore the fine details and produce a high-resolution shadow-free image. DRNet shares a similar architecture to LSRNet but omits the self-attention and CBAM layers to maintain its lightweight nature.

To complement the shadow removal process, the authors also propose a lightweight shadow detector network. This network is designed for efficiency, using a MobileNetV2 backbone and an ASPP module, making it suitable for mobile deployment.

Datasets

The paper utilizes and introduces the following datasets:

  • ISTD (Image Shadow Triplets Dataset): A standard, large-scale dataset for shadow removal, containing shadow images, shadow masks, and shadow-free images at a resolution of 640x480.
  • ISTD-HQ: A high-resolution version of the ISTD dataset created by upscaling the original images to 2560x1920 using a super-resolution network. This dataset was created to test the proposed framework's applicability to high-resolution inference.
  • Shadow Food-HQ (SFHQ): A newly constructed dataset of high-resolution food images captured at 12MP (4032x3024) resolution. It contains 14520 shadow triplets with manually annotated shadow masks, focusing on externally cast shadows.

Key Insights and Ideas

  • Dual-stage processing for high-resolution: The core idea of processing at low-resolution first to capture global context and then refining at high-resolution is crucial for handling large images effectively. This breaks down the complexity of the problem.
  • Lightweight architectures are key: The paper emphasizes the importance of designing efficient networks (LSRNet and DRNet) that can be deployed on resource-constrained devices, a significant departure from many existing heavy models.
  • Self-attention for global context: Incorporating self-attention in the bottleneck of LSRNet is vital for capturing long-range dependencies, which are essential for consistent shadow relighting.
  • DRNet as a detail refiner: Leveraging the high-resolution input as a guide for DRNet is an efficient way to restore details without requiring the entire network to process at high resolution from the start.
  • Mobile deployment is achievable: Demonstrating that the entire pipeline (detection + removal) can run on a modern smartphone in under 3 seconds highlights the practical impact of the proposed lightweight designs.

Key Results

  • State-of-the-art performance: SHARDS achieves state-of-the-art results on the ISTD dataset with significantly fewer network parameters and computations compared to existing methods.
  • High-resolution effectiveness: Qualitative comparisons show that SHARDS maintains shadow removal quality at progressively higher resolutions (up to 2048x2048), while other methods degrade.
  • Computational efficiency: LSRNet has 65x fewer parameters than existing methods. The combined SHARDS architecture is computationally efficient, with the DRNet being particularly lightweight.
  • On-device performance: The proposed architecture can perform shadow removal on a 12MP image on a modern smartphone in approximately 2.4 seconds.
  • Shadow detection: The proposed lightweight shadow detector achieves comparable performance to existing methods while being more parameter-efficient.

Limitations and Future Work

  • Paired dataset requirement: Like many existing methods, SHARDS relies on paired shadow and shadow-free images for training, which can limit its generalization to unpaired scenarios. The authors suggest that their ideas are applicable to unpaired methods as well.
  • Dataset simplicity: The existing datasets are considered simplistic, and real-world shadows can be more complex, including self-cast shadows, which might require more robust detection and removal.
  • Future work: The authors suggest exploring robustness to self-cast shadows and developing methods that are less reliant on paired datasets.

Summary updated

BibTeX
@inproceedings{sen2023shards,
  author = {Mrinmoy Sen and Sai Pradyumna Chermala and Nazrinbanu Nurmohammad Nagori and Venkat Peddigari and Praful Mathur and B. H. Pawan Prasad and Moonhwan Jeong},
  title = {{SHARDS: Efficient Shadow Removal Using Dual Stage Network for High-Resolution Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure