TL;DR
This paper introduces LRA&LDRA, novel modules that guide existing shadow removal models to focus on shadow regions, leading to improved accuracy and efficiency. They also present PITSA, a large-scale synthetic dataset for shadow detection and removal, which significantly boosts model performance when used for pre-training.
What Problem Does This Paper Address?
Shadows in images can degrade the performance of various computer vision tasks by obscuring details or creating false objects. While shadow detection and removal techniques exist, current methods often struggle with two main issues: inefficient use of model capacity by reconstructing entire images (including non-shadow regions) and the limited size and diversity of available training datasets. Existing residual prediction methods, which aim to focus on the difference between shadowed and shadow-free images, offer some improvement but still fall short of optimal results.
Methods
The core contribution is the LRA (Learnable Residual Attention) and LDRA (Learnable Dense Reconstruction Attention) modules. These modules are designed to be plugged into existing shadow removal models (SRMs). Unlike previous approaches that might reconstruct the whole image or simply predict residuals, LRA&LDRA operate on both the input and output of the SRM. Their key functions are:
- Guiding Focus: They direct the SRM to concentrate its reconstruction efforts specifically on shadow regions, preventing wasted capacity on non-shadow areas.
- Blending and Color Correction: They assist in refining the output by performing blending and color correction, leading to more natural-looking results.
The authors propose a specific implementation of LRA&LDRA using a coordinate attention mechanism, which is efficient and has a strong spatial component. They also introduce a novel dataset generation pipeline to create PITSA. This pipeline involves extracting shadow-free patches from real images and then synthesizing shadows onto them, allowing for a much larger and more diverse dataset than previously available.
Datasets
- ISTD [65]: Used for training and evaluation of shadow detection and removal models.
- SRD [54]: Used for training and evaluation of shadow removal models.
- PITSA (Proposed): A newly generated dataset consisting of 172,539 triplets (shadow image, shadow-free image, shadow mask). It's created using a pipeline that extracts shadow-free patches and synthesizes shadows, aiming for greater scale and diversity than existing datasets.
Key Insights and Ideas
- Shadow Removal is Localized Translation: The paper frames shadow removal as a localized image-to-image translation problem, emphasizing that only the shadow regions need significant alteration.
- Residuals Need Guidance: Simply predicting residuals isn't enough; LRA&LDRA provide the necessary attention mechanism to guide the model on how to predict and refine these residuals effectively.
- Dataset Scale Matters: Pre-training on the large-scale PITSA dataset significantly boosts performance across various models, highlighting the importance of data volume and diversity for this task.
- Joint Training Benefits: LRA&LDRA facilitate better gradient propagation between the shadow detection and removal models, leading to improved detection accuracy.
Key Results
- Improved Accuracy: LRA&LDRA modules improve shadow removal accuracy by up to 20% and detection accuracy by up to 10% when integrated into existing SRMs.
- Efficiency: The modules add minimal overhead, requiring less than 1MB of memory and only a few milliseconds of runtime.
- State-of-the-Art Performance: When combined with pre-training on PITSA and a lightweight architecture, LRA&LDRA outperform state-of-the-art methods on benchmark datasets (ISTD and SRD) in both removal (e.g., +0.7 MAE on all regions) and detection (e.g., +0.1 BER).
- Faster and Smaller: The proposed lightweight model with LRA&LDRA is significantly faster (+5%) and consumes 150x less memory than some state-of-the-art methods.
- PITSA Impact: Pre-training on PITSA leads to substantial improvements, such as +2 MAE on shadow regions for removal and +0.1 BER for detection on ISTD.
Limitations and Future Work
- Shadow Detector Dependency: The PITSA dataset generation relies on a pre-trained shadow detection model. Errors from this detector can introduce noise into the generated dataset, though the authors argue that the scale of PITSA mitigates this.
- Future Work: The authors suggest that the shadow detector used in their pipeline could be improved, and their LRA&LDRA framework can be extended to other vision tasks. They also highlight the flexibility of their approach, allowing for different implementations of LRA and LDRA.
Summary updated