Adaptive Illumination Mapping for Shadow Detection in Raw Images

Jiayu Sun, Ke Xu, Youwei Pang, Lihe Zhang, Huchuan Lu, Gerhard P. Hancke, Rynson W. H. Lau

Abstract

Shadow detection in sRGB images is often hindered by the Image Signal Processor (ISP), which compresses global contrast to preserve local details. This paper proposes a novel framework that detects shadows directly from RAW images using an Adaptive Illumination Mapping (AIM) module. By learning a many-to-many mapping that preserves multi-scale contrast and incorporating a shadow-aware feedback mechanism, the proposed method significantly outperforms existing sRGB-based detectors.

Introduction

Shadow detection is a critical component of many computer vision pipelines, yet it remains a challenging task. The primary difficulty lies in distinguishing shadows from dark, non-shadow objects, as both often exhibit similar local intensities. Traditional methods typically operate on sRGB images, which have already undergone significant processing within the camera's Image Signal Processor (ISP). This paper argues that the ISP itself is a major bottleneck and proposes a solution that leverages RAW image data directly.

The ISP Bottleneck

When a camera captures a scene, it records high-dynamic-range (HDR) RAW data. The internal ISP then converts this into a standard sRGB image. To create a visually pleasing result, ISPs often perform a "many-to-one" mapping that compresses global contrast to emphasize local details. While this enhances image aesthetics, it destroys the global intensity relationships that are essential for robust shadow detection. Consequently, shadows and dark surfaces become nearly indistinguishable in the sRGB domain.

Methodology: Adaptive Illumination Mapping

To overcome this limitation, the researchers developed a network architecture that bypasses the traditional ISP. The core of the approach is the Adaptive Illumination Mapping (AIM) module. Instead of a single output, the AIM module learns a "many-to-many" mapping, projecting linear RAW data into multiple sRGB images representing different intensity ranges. This ensures that contrast is preserved across multiple scales.

The system also includes a Shadow-Aware Feedback Mechanism. During training, initial shadow predictions are fed back into the AIM module. This forces the module to adaptively "re-expose" the generated images to maximize the visual difference between shadow and non-shadow regions, effectively creating a task-specific illumination enhancement.

The SARA Dataset

Because existing shadow detection datasets lack RAW data, the authors introduced the ShAdow RAw (SARA) dataset. It contains 7,019 pairs of RAW images and high-quality shadow masks, covering a wide variety of foreground objects and background scenes.

An example input image from the SARA dataset featuring a complex shadow on a brick walkway.

Key Results

The proposed method, also named SARA, achieves state-of-the-art performance. On its namesake dataset, it reached a Balanced Error Rate (BER) of 2.61, a significant improvement over the best sRGB-based methods (e.g., MTMT at 3.65) and even methods adapted for RAW data.

The corresponding shadow mask produced by the SARA model, demonstrating precise boundary detection.

Qualitatively, the model excels in scenarios with high global contrast but low local contrast. Furthermore, the model is highly efficient, with an inference time of approximately 0.03 seconds on a standard GTX 3090 GPU, making it suitable for practical applications.

Conclusion

By shifting the focus from processed sRGB images to raw sensor data, this research highlights the importance of data integrity in low-level vision tasks. The adaptive illumination mapping approach provides a robust framework for handling high-dynamic-range scenes, with potential applications extending beyond shadow detection to areas like low-light enhancement and HDR imaging.

Summary updated

Abstract

Shadow detection methods rely on multi-scale contrast, especially global contrast, information to locate shadows correctly. However, we observe that the camera image signal processor (ISP) tends to preserve more local contrast information by sacrificing global contrast information during the raw-to-sRGB conversion process. This often causes existing methods to fail in scenes with high global contrast but low local contrast in shadow regions. In this paper, we propose a novel method to detect shadows from raw images. Our key idea is that instead of performing a many-to-one mapping like the ISP process, we can learn a many-to-many mapping from the high dynamic range raw images to the sRGB images of different illumination, which is able to preserve multi-scale contrast for accurate shadow detection. To this end, we first construct a new shadow dataset with ~ 7000 raw images and shadow masks. We then propose a novel network, which includes a novel adaptive illumination mapping (AIM) module to project the input raw images into sRGB images of different intensity ranges and a shadow detection module to leverage the preserved multi-scale contrast information to detect shadows. To learn the shadow-aware adaptive illumination mapping process, we propose a novel feedback mechanism to guide the AIM during training. Experiments show that our method outperforms state- of-the-art shadow detectors. Code and dataset are available at https://github.com/jiayusun/SARA.
BibTeX
@inproceedings{sun2023adaptive,
  author = {Jiayu Sun and Ke Xu and Youwei Pang and Lihe Zhang and Huchuan Lu and Gerhard P. Hancke and Rynson W. H. Lau},
  title = {{Adaptive Illumination Mapping for Shadow Detection in Raw Images}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01167},
}
Figure