Abstract
Cloud removal (CR) remains a challenging task in remote sensing image processing. Although diffusion models (DM) exhibit strong generative capabilities, their direct applications to CR are suboptimal, as they generate cloudless images from random noise, ignoring inherent information in cloudy inputs. To overcome this drawback, we develop a new CR model EMRDM based on mean-reverting diffusion models (MRDMs) to establish a direct diffusion process between cloudy and cloudless images. Compared to current MRDMs, EMRDM offers a modular framework with updatable modules and an elucidated design space, based on a reformulated forward process and a new ordinary differential equation (ODE)-based backward process. Leveraging our framework, we redesign key MRDM modules to boost CR performance, including restructuring the denoiser via a preconditioning technique, reorganizing the training process, and improving the sampling process by introducing deterministic and stochastic samplers. To achieve multi-temporal CR, we further develop a denoising network for simultaneously denoising sequential images. Experiments on mono-temporal and multi-temporal datasets demonstrate the superior performance of EMRDM.
Introduction
Cloud cover is a persistent obstacle in remote sensing, obscuring approximately 67% of the Earth's surface and hindering the utility of satellite imagery for applications like environmental monitoring and agriculture. Traditional approaches, such as Generative Adversarial Networks (GANs), often suffer from training instability. While Diffusion Models (DMs) have recently emerged as a powerful alternative due to their generative fidelity, standard implementations typically initiate the restoration process from pure Gaussian noise. This "blank canvas" approach neglects the structural information present in the cloudy input, frequently resulting in "hallucinated" details that lack pixel-level consistency with the ground truth.
Methodology: The EMRDM Framework
To address these limitations, we propose the Elucidated Mean-Reverting Diffusion Model (EMRDM). The core innovation lies in transitioning from standard diffusion to a mean-reverting process that treats the cloudy image as a structural anchor.
Mean-Reverting Diffusion
Unlike standard DMs that diffuse an image toward zero-mean Gaussian noise, EMRDM utilizes a Mean-Reverting Stochastic Differential Equation (SDE). In this process, the target clear image is diffused toward the cloudy image (the "mean"). This ensures that the starting point for the reverse restoration process is physically and structurally anchored to the actual scene, preserving landscape fidelity and reducing distortion.
Elucidated Design Space and Preconditioning
EMRDM introduces an "elucidated" design space, decoupling the noise schedule from the network architecture. This modularity allows for independent updates to the forward SDE and the backward Ordinary Differential Equation (ODE) processes. To ensure training stability and prevent gradient issues, we implement a preconditioning technique. This scales inputs and outputs to maintain unit variance, a critical factor for the performance of deep denoising networks.
Multi-temporal Denoising
For multi-temporal datasets, where a sequence of images of the same location is available, we developed a denoising network featuring Temporal Fusion Self-Attention (TFSA). TFSA enables the model to dynamically identify and leverage cloud-free pixels from different time steps to reconstruct obscured areas with high fidelity.
Experimental Results
EMRDM was evaluated against state-of-the-art (SOTA) methods on four benchmarks: CUHK-CR1, CUHK-CR2, SEN12MS-CR, and Sen2_MTC_New.
- Quantitative Performance: On the SEN12MS-CR dataset, EMRDM achieved a Peak Signal-to-Noise Ratio (PSNR) of 32.14 and a Structural Similarity Index (SSIM) of 0.924, outperforming previous diffusion-based models like DiffCR.
- Qualitative Quality: Visual inspections reveal that EMRDM restores significantly finer textures and sharper outlines compared to GAN-based methods, which often produce overly smooth or "plastic" artifacts.
- Efficiency: A key finding is the model's efficiency; EMRDM achieves superior restoration quality in as few as 5 sampling steps, a substantial improvement over the hundreds of steps typically required by standard generative DMs.
Conclusion
EMRDM represents a significant advancement in cloud removal by integrating mean-reverting logic with an elucidated, modular design space. By anchoring the diffusion process to the cloudy input and employing advanced preconditioning and temporal attention, the model achieves superior structural fidelity and restoration quality. Future work will focus on reducing the computational overhead further and improving generalization to diverse "in-the-wild" atmospheric conditions.
Summary updated