TL;DR
This paper introduces a novel semi-supervised learning framework that leverages unlabeled real-world rainy images to significantly improve the performance of image deraining models. By modeling latent space representations with Gaussian Processes, the method generates "pseudo-ground-truth" to guide the network, leading to better generalization than methods trained solely on synthetic data.
What Problem Does This Paper Address?
Image deraining, the process of removing rain streaks from photographs, is crucial for many computer vision applications like object detection and surveillance. While deep learning models have achieved impressive results, they typically require large amounts of fully labeled data (rainy images paired with their clean counterparts). Obtaining such real-world labeled datasets is extremely challenging. Consequently, most existing methods rely on synthetically generated data, which often leads to a performance gap when applied to real-world images due to domain shift. This paper tackles this problem by proposing a method that effectively utilizes readily available unlabeled real-world rainy images to enhance the generalization capabilities of deraining networks.
Methods
The core of the proposed method is a Gaussian Process (GP)-based semi-supervised learning (SSL) framework. The training process is iterative, alternating between learning from labeled synthetic data and unlabeled real-world data.
- Labeled Training Phase: The network is trained using a supervised loss function that combines L1 loss and perceptual loss between the predicted derained image and the ground-truth clean image. Crucially, the intermediate latent space vectors from this phase are stored.
- Unlabeled Training Phase: For unlabeled real-world images, the method aims to supervise the intermediate latent space. This is achieved by:
- Modeling Latent Space with GPs: The stored latent space vectors from labeled data, along with the latent vectors of the current unlabeled image, are modeled using Gaussian Processes. This allows for the estimation of a "pseudo-ground-truth" latent vector for the unlabeled image.
- Feature-Level GP Modeling: A key improvement over prior work is modeling GPs at the feature map level within the latent space. This accounts for the potential independence of different feature maps, leading to more accurate pseudo-ground-truth generation.
- Uncertainty-Aware Pseudo-GT: The GP formulation naturally provides uncertainty estimates. The method uses these uncertainties to re-weight the loss function, ensuring that only confident pseudo-ground-truth samples contribute significantly to training.
- Unsupervised Loss: The error between the predicted latent vector and the generated pseudo-ground-truth latent vector is minimized, guiding the network to better adapt to the unlabeled data distribution.
The overall network architecture is a U-Net-style encoder-decoder, with Res2Blocks as building blocks.
Datasets
The paper utilizes several datasets for training and evaluation:
- DIDMDN: A synthetic dataset with varying rain densities, used as a labeled source dataset.
- DDN dataset: Another synthetic dataset used for training and testing.
- Rain800: A synthetic dataset for training and testing.
- Rain200L: A synthetic dataset used for cross-domain experiments.
- DDN-SIRR dataset: This dataset is specifically constructed for evaluating semi-supervised deraining methods. It includes a labeled synthetic training set and an unlabeled real-world training set, along with dense and sparse rain streak test sets.
The paper also mentions using unlabeled rainy images from sources like [60], [11], [18], and Google image search for the DDN-SIRR unlabeled set.
Key Insights and Ideas
- Leveraging Unlabeled Real-World Data is Key: The most significant insight is that incorporating unlabeled real-world data, despite its inherent noise and domain shift, is crucial for achieving robust and generalizable image deraining performance.
- Gaussian Processes for Latent Space Modeling: Using GPs to model the latent space provides a principled way to generate reliable pseudo-ground-truth for unlabeled data, capturing the underlying data distribution.
- Feature-Level GP Modeling is Superior: Modeling GPs at the feature map level within the latent space, rather than the entire latent vector, significantly improves the accuracy of pseudo-ground-truth generation by respecting the independence of individual feature representations.
- Uncertainty-Guided Supervision: Utilizing GP uncertainty estimates to weight the unsupervised loss is an effective strategy to prevent noisy or inaccurate pseudo-ground-truth from degrading performance.
- Bridging the Synthetic-to-Real Gap: The proposed method effectively bridges the domain gap between synthetic and real-world rainy images, leading to substantial performance gains.
Key Results
- Improved Generalization: The proposed method (Syn2Real++) consistently outperforms methods trained only on synthetic data and existing semi-supervised methods like SIRR [19] across various datasets (DDN-SIRR, Rain800, Rain200L, DDN).
- Quantitative Gains: On the DDN-SIRR synthetic test set, the proposed method achieved PSNR gains of up to 1.25 dB over the labeled-only baseline and outperformed SIRR by a significant margin. On the real-world test set, it achieved lower NIQE and BRISQUE scores (indicating better quality) compared to SIRR and its previous version (Syn2Real [20]).
- Cross-Domain Performance: The method demonstrates strong performance in cross-domain scenarios, where it leverages unlabeled data from one domain to improve deraining on another, outperforming previous approaches.
- Qualitative Improvements: Visual results show that the proposed method produces cleaner and sharper derained images, with fewer residual rain streaks, especially in challenging scenarios.
Limitations and Future Work
- Computational Cost: While not explicitly detailed as a limitation, GP inference can be computationally intensive, especially with large datasets. The paper mentions using nearest neighbors to mitigate this.
- Hyperparameter Sensitivity: The performance is influenced by hyperparameters like $\lambda_{unsup}$ and $N_n$. The paper includes ablation studies to find optimal values, but careful tuning might still be required.
- Future Work: The paper suggests exploring different kernel functions for GPs and investigating more advanced uncertainty estimation techniques. The authors also implicitly suggest that further research into bridging domain gaps in deraining is valuable.
Summary updated