TL;DR
This paper introduces S2-ShadowNet, a novel weakly supervised approach for remote sensing shadow removal that leverages the complementary information from visible and infrared imagery. By mapping features to a spherical space and employing specialized loss functions, S2-ShadowNet effectively aligns shared illumination information and separates private textural details, achieving state-of-the-art results.
What Problem Does This Paper Address?
Shadows in remote sensing imagery pose a significant challenge for various downstream computer vision tasks like object detection and semantic segmentation. These shadows are characterized by extremely low illumination, making it difficult to recover the underlying surface information. While infrared imagery is known to be robust to illumination changes and can provide valuable cues, existing methods often fail to effectively integrate information from both visible and infrared modalities. This lack of cross-modal collaboration leads to degraded performance. Furthermore, obtaining perfectly paired shadow and shadow-free images for training is practically difficult and expensive, necessitating weakly supervised approaches.
Methods
The proposed S2-ShadowNet tackles shadow removal through a three-stage process:
- Modal Translation: An auxiliary model (pretrained U2Fusion) is used to generate realistic infrared images from visible ones. This helps bridge the domain gap and provides synthetic infrared data for training.
- Shadow Generation: An adversarial learning strategy is employed to generate pseudo-shadows for shadow-free regions. This is crucial for creating training signals in a weakly supervised setting, as it allows the network to learn from self-generated shadow-like artifacts.
- Cross-Modal Spherical Aggregation: This is the core innovation. Features extracted by Swin Transformers from both visible and infrared images are projected into a spherical space. This spherical representation offers advantages in domain adaptation by regularizing distances between features. Within this space, two key loss functions are applied:
- Orthogonality Loss: Encourages the separation of private features (e.g., texture in visible, thermal information in infrared) by ensuring their representations are orthogonal.
- Similarity Loss: Promotes the alignment of shared features (illumination components) by encouraging their representations to be similar.
This spherical aggregation and dual-loss strategy allows for implicit reciprocity between modalities, enabling effective shadow removal without requiring strictly paired data. The final shadow-free image is reconstructed by combining the full visible features with the aligned shared infrared features.
Datasets
The paper introduces a new large-scale weakly supervised shadow removal benchmark called WSSR. This dataset comprises 4000 shadow images with corresponding shadow masks, collected from diverse geographical locations.
In addition to WSSR, the authors also evaluate their method on two existing benchmarks:
- URSSR: Contains unpaired shadow and shadow-free images.
- UAV-SC: A benchmark with 6954 shadow image pairs and corresponding ground truths.
Key Insights and Ideas
- Infrared as an Illumination Prior: The paper highlights the significant potential of infrared imagery as a complementary source of illumination information that is often missing or distorted in visible-only shadow removal methods.
- Spherical Space for Cross-Modal Alignment: Mapping features to a spherical manifold is a novel way to handle domain shifts and facilitate the alignment and separation of multi-modal representations. The inherent geometric properties of the sphere help regularize feature distances.
- Weakly Supervised Learning via Self-Generation: The use of a shadow generation strategy to create training signals from shadow-free regions is a clever way to overcome the challenge of acquiring paired data, making the approach more practical.
- Orthogonality and Similarity for Feature Decomposition: The combination of orthogonality and similarity losses in the spherical space provides a principled way to disentangle shared and private information across modalities, leading to more robust shadow removal.
Key Results
- State-of-the-Art Performance: S2-ShadowNet consistently outperforms existing traditional, unsupervised, and weakly supervised shadow removal methods on the WSSR, URSSR, and UAV-SC benchmarks, both qualitatively and quantitatively.
- Quantitative Improvements: On the UAV-SC benchmark, S2-ShadowNet achieves competitive results, even comparable to some fully supervised methods, while significantly outperforming others. On WSSR and URSSR, it shows substantial percentage gains in VNM and Entropy scores compared to top-performing methods like LG-ShadowNet.
- Qualitative Superiority: Visual comparisons demonstrate that S2-ShadowNet effectively removes shadows while preserving color consistency and rich details, avoiding artifacts and distortions seen in other methods. It handles complex shadow scenarios, including varying intensities and overlaps, more effectively.
Limitations and Future Work
- Fully Supervised vs. Weakly Supervised: While S2-ShadowNet achieves strong results in a weakly supervised setting, fully supervised methods still hold an advantage in full-reference evaluations due to the availability of ground truth.
- Ablation Studies: The paper includes extensive ablation studies that analyze the contribution of each component (modal translation, spherical transform, and individual loss functions). These studies confirm the effectiveness of each element and their synergistic effect in the full model.
- Future Directions: The authors suggest that their work opens up avenues for further research in multi-modal collaboration for various remote sensing tasks. The contribution of a large-scale weakly supervised benchmark also facilitates future development in this area.
Summary updated