Semi-Supervised Image Deraining Using Gaussian Processes

Rajeev Yasarla, Vishwanath A. Sindagi, Vishal M. Patel

TL;DR

This paper introduces a novel semi-supervised learning framework that leverages unlabeled real-world rainy images to significantly improve the performance of image deraining models. By modeling latent space representations with Gaussian Processes, the method generates "pseudo-ground-truth" to guide the network, leading to better generalization than methods trained solely on synthetic data.

What Problem Does This Paper Address?

Image deraining, the process of removing rain streaks from photographs, is crucial for many computer vision applications like object detection and surveillance. While deep learning models have achieved impressive results, they typically require large amounts of fully labeled data (rainy images paired with their clean counterparts). Obtaining such real-world labeled datasets is extremely challenging. Consequently, most existing methods rely on synthetically generated data, which often leads to a performance gap when applied to real-world images due to domain shift. This paper tackles this problem by proposing a method that effectively utilizes readily available unlabeled real-world rainy images to enhance the generalization capabilities of deraining networks.

Methods

The core of the proposed method is a Gaussian Process (GP)-based semi-supervised learning (SSL) framework. The training process is iterative, alternating between learning from labeled synthetic data and unlabeled real-world data.

  1. Labeled Training Phase: The network is trained using a supervised loss function that combines L1 loss and perceptual loss between the predicted derained image and the ground-truth clean image. Crucially, the intermediate latent space vectors from this phase are stored.
  2. Unlabeled Training Phase: For unlabeled real-world images, the method aims to supervise the intermediate latent space. This is achieved by:
  3. Modeling Latent Space with GPs: The stored latent space vectors from labeled data, along with the latent vectors of the current unlabeled image, are modeled using Gaussian Processes. This allows for the estimation of a "pseudo-ground-truth" latent vector for the unlabeled image.
  4. Feature-Level GP Modeling: A key improvement over prior work is modeling GPs at the feature map level within the latent space. This accounts for the potential independence of different feature maps, leading to more accurate pseudo-ground-truth generation.
  5. Uncertainty-Aware Pseudo-GT: The GP formulation naturally provides uncertainty estimates. The method uses these uncertainties to re-weight the loss function, ensuring that only confident pseudo-ground-truth samples contribute significantly to training.
  6. Unsupervised Loss: The error between the predicted latent vector and the generated pseudo-ground-truth latent vector is minimized, guiding the network to better adapt to the unlabeled data distribution.

The overall network architecture is a U-Net-style encoder-decoder, with Res2Blocks as building blocks.

Datasets

The paper utilizes several datasets for training and evaluation:

  • DIDMDN: A synthetic dataset with varying rain densities, used as a labeled source dataset.
  • DDN dataset: Another synthetic dataset used for training and testing.
  • Rain800: A synthetic dataset for training and testing.
  • Rain200L: A synthetic dataset used for cross-domain experiments.
  • DDN-SIRR dataset: This dataset is specifically constructed for evaluating semi-supervised deraining methods. It includes a labeled synthetic training set and an unlabeled real-world training set, along with dense and sparse rain streak test sets.

The paper also mentions using unlabeled rainy images from sources like [60], [11], [18], and Google image search for the DDN-SIRR unlabeled set.

Key Insights and Ideas

  • Leveraging Unlabeled Real-World Data is Key: The most significant insight is that incorporating unlabeled real-world data, despite its inherent noise and domain shift, is crucial for achieving robust and generalizable image deraining performance.
  • Gaussian Processes for Latent Space Modeling: Using GPs to model the latent space provides a principled way to generate reliable pseudo-ground-truth for unlabeled data, capturing the underlying data distribution.
  • Feature-Level GP Modeling is Superior: Modeling GPs at the feature map level within the latent space, rather than the entire latent vector, significantly improves the accuracy of pseudo-ground-truth generation by respecting the independence of individual feature representations.
  • Uncertainty-Guided Supervision: Utilizing GP uncertainty estimates to weight the unsupervised loss is an effective strategy to prevent noisy or inaccurate pseudo-ground-truth from degrading performance.
  • Bridging the Synthetic-to-Real Gap: The proposed method effectively bridges the domain gap between synthetic and real-world rainy images, leading to substantial performance gains.

Key Results

  • Improved Generalization: The proposed method (Syn2Real++) consistently outperforms methods trained only on synthetic data and existing semi-supervised methods like SIRR [19] across various datasets (DDN-SIRR, Rain800, Rain200L, DDN).
  • Quantitative Gains: On the DDN-SIRR synthetic test set, the proposed method achieved PSNR gains of up to 1.25 dB over the labeled-only baseline and outperformed SIRR by a significant margin. On the real-world test set, it achieved lower NIQE and BRISQUE scores (indicating better quality) compared to SIRR and its previous version (Syn2Real [20]).
  • Cross-Domain Performance: The method demonstrates strong performance in cross-domain scenarios, where it leverages unlabeled data from one domain to improve deraining on another, outperforming previous approaches.
  • Qualitative Improvements: Visual results show that the proposed method produces cleaner and sharper derained images, with fewer residual rain streaks, especially in challenging scenarios.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, GP inference can be computationally intensive, especially with large datasets. The paper mentions using nearest neighbors to mitigate this.
  • Hyperparameter Sensitivity: The performance is influenced by hyperparameters like $\lambda_{unsup}$ and $N_n$. The paper includes ablation studies to find optimal values, but careful tuning might still be required.
  • Future Work: The paper suggests exploring different kernel functions for GPs and investigating more advanced uncertainty estimation techniques. The authors also implicitly suggest that further research into bridging domain gaps in deraining is valuable.

Summary updated

Abstract

Recent CNN-based methods for image deraining have achieved excellent performance in terms of reconstruction error as well as visual quality. However, these methods are limited in the sense that they can be trained only on fully labeled data. Due to various challenges in obtaining real world fully-labeled image deraining datasets, existing methods are trained only on synthetically generated data and hence, generalize poorly to real-world images. The use of real-world data in training image deraining networks is relatively less explored in the literature. We propose a Gaussian Process-based semi-supervised learning framework which enables the network in learning to derain using synthetic dataset while generalizing better using unlabeled real-world images. More specifically, we model the latent space vectors of unlabeled data using Gaussian Processes, which is then used to compute pseudo-ground-truth for supervising the network on unlabeled data. The pseudo ground-truth is further used to supervise the network at the intermediate level for the unlabeled data. Through extensive experiments and ablations on several challenging datasets (such as Rain800, Rain200L and DDN-SIRR), we show that the proposed method is able to effectively leverage unlabeled data thereby resulting in significantly better performance as compared to labeled-only training. Additionally, we demonstrate that using unlabeled real-world images in the proposed GP-based framework results in superior performance as compared to the existing methods. Code is available at: https://github.com/rajeevyasarla/Syn2Real.
BibTeX
@article{yasarla2021semi,
  author = {Rajeev Yasarla and Vishwanath A. Sindagi and Vishal M. Patel},
  title = {{Semi-Supervised Image Deraining Using Gaussian Processes}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=JF0qTeL4nU},
}
Figure