TL;DR
This paper introduces SFCRFormer, a novel Transformer-based network for removing clouds from optical remote sensing images. By effectively combining spatial and frequency domain information through specialized attention modules, SFCRFormer achieves superior cloud removal performance, preserving image details and reducing artifacts compared to existing methods.
What Problem Does This Paper Address?
Optical remote sensing images are invaluable for Earth observation, but clouds frequently obscure crucial information, degrading image quality and hindering downstream applications like land cover classification and water body extraction. Existing cloud removal techniques often struggle to effectively integrate spatial and frequency features, leading to issues like edge blurring and color distortions. This paper aims to bridge this gap by proposing a method that synergistically leverages both spatial and frequency information for more accurate and visually appealing cloud removal.
Methods
The core innovation is the Spatial-Frequency Combined Transformer (SFCT) block. This block is built upon two key modules:
- Dual-Branch Spatial Attention (DBSA): This module enhances spatial feature representation by decoupling spatial and channel dependencies. It uses parallel feature refinement paths, which is an improvement over traditional single-branch attention mechanisms for maintaining image structure.
- Frequency Self-Attention (FreSA): This module utilizes the Fast Fourier Transform (FFT) to move features into the frequency domain. By analyzing frequency differences between cloud and ground regions, it enables precise cloud detection and fine-grained removal.
To further refine features, the authors introduce the Dual-Domain Feed-Forward Network (DDFFN). This network combines multi-scale convolutions for local detail enhancement with frequency transformation for global structural optimization.
Finally, a composite loss function is employed, combining Charbonnier loss (for robustness and detail preservation) and Structural Similarity Index (SSIM) loss (for perceptual quality and structural fidelity), to optimize training and balance pixel-level accuracy with structural coherence.
Datasets
The proposed SFCRFormer was trained and evaluated on the following publicly available datasets:
- RICE Dataset: This dataset consists of two subsets, RICE1 (thin cloud) and RICE2 (thick cloud), containing pairs of cloudy images and ground truth.
- T-Cloud Dataset: This is a large-scale thin cloud dataset collected from Landsat 8.
The paper does not introduce any newly created benchmark datasets.
Key Insights and Ideas
- Synergy of Spatial and Frequency Domains: The paper strongly advocates for the synergistic integration of spatial and frequency information, arguing that neither domain alone is sufficient for optimal cloud removal. The SFCT block is designed to facilitate this interaction.
- Frequency Domain for Cloud Discrimination: Leveraging the distinct frequency characteristics of clouds (low-frequency dominance) and clear land cover (rich textures, high-frequency components) is a powerful strategy for precise cloud identification and removal.
- Dual-Branch Attention for Robustness: The DBSA module's parallel branches for spatial and channel attention offer a more robust way to capture image structure compared to single-branch approaches.
- Frequency Transformation for Detail Preservation: The FreSA module's ability to operate in the frequency domain allows it to better distinguish between cloud-obscured areas and underlying details, leading to finer reconstruction.
- Composite Loss for Balanced Optimization: Combining pixel-level accuracy (Charbonnier loss) with perceptual and structural quality (SSIM loss) is crucial for achieving visually pleasing and accurate results.
Key Results
The SFCRFormer demonstrated state-of-the-art performance across all tested datasets (RICE1, RICE2, and T-Cloud).
- Quantitative Improvements: The method consistently outperformed existing methods (including GAN-based and Transformer-based approaches like SpA GAN, AMGAN, CVAE, Restormer, and TCME) in metrics such as PSNR, SSIM, MAE, and RMSE. For instance, on the RICE1 dataset, SFCRFormer achieved a PSNR of 37.3512 and an SSIM of 0.9699, outperforming the second-best method (Restormer) by significant margins.
- Qualitative Improvements: Visual comparisons showed that SFCRFormer generated cloud-free images with fewer artifacts, richer details, and better color fidelity. It excelled at recovering fine-grained structural details and edges, even in challenging thick cloud scenarios. GAN-based methods often produced blurry images with artifacts, while other Transformer-based methods, though better, still struggled with intricate details.
Limitations and Future Work
- Limitations: The paper does not explicitly detail significant limitations of the proposed method. However, as with most deep learning models, performance might be sensitive to the diversity and quality of training data.
- Future Work: The authors plan to extend SFCRFormer to SAR and optical image fusion for cloud removal. They believe that by leveraging the complementary information from these two modalities, they can achieve even more precise and robust cloud removal results, expanding the framework's applicability to a wider range of remote sensing scenarios.
Summary updated