Unsupervised Restoration of Weather-affected Images using Deep Gaussian Process-based CycleGAN

Rajeev Yasarla, Vishwanath A. Sindagi, Vishal M. PatelarXiv 2022 · OpenReview

TL;DR

This paper tackles the challenge of restoring images degraded by weather (rain, haze, snow) without requiring paired clean/degraded images for training. They achieve this by introducing novel "Deep Gaussian Process-based" losses to the CycleGAN framework, enabling effective learning from real-world, unpaired data and outperforming existing unsupervised methods.

What Problem Does This Paper Address?

Restoring images affected by weather phenomena like rain, fog, or snow is crucial for improving image quality and the performance of downstream computer vision tasks. Traditional methods often rely on fully supervised learning, which necessitates paired datasets of clean and degraded images. However, collecting such paired data in real-world weather conditions is extremely difficult, leading most existing methods to train on synthetic data. This synthetic-to-real domain shift often results in poor performance on actual real-world images. Unsupervised methods exist, but they often struggle to achieve high-quality restorations or are tailored to specific weather conditions. This paper aims to bridge this gap by enabling effective unsupervised restoration using real-world, unpaired data.

Methods

The core of the proposed method is an extension of the CycleGAN architecture. CycleGAN is an unsupervised image-to-image translation framework that uses two generators (one for mapping from weather-affected to clean, and another for the reverse) and enforces cycle-consistency. The key innovation here is the introduction of new loss functions that provide additional supervision in the latent space.

These new losses are derived using Deep Gaussian Processes (GPs). The idea is to model the relationship between latent space embeddings of original clean images and the latent embeddings of the predicted clean images generated by the CycleGAN. By conditioning on the original clean image embeddings, the GP can predict "pseudo-labels" for the generated embeddings. Crucially, GPs also provide an estimate of uncertainty. This uncertainty information is used to weight the pseudo-labels, ensuring that only confident predictions contribute to the loss. This prevents the network from learning from noisy or incorrect translations, especially during the early stages of training. The authors formulate these losses for both the forward (weather-affected to clean) and reverse (clean to weather-affected) mapping networks.

Datasets

The paper evaluates their method on several standard datasets for different weather restoration tasks:

  • De-raining: Rain100L, Rain100H, DIDMDN, SPANet.
  • De-hazing: SOTS-I, SOTS-O (from RESIDE dataset), RTTS.
  • De-snowing: Snow100k.

They also use the Cityscapes dataset for evaluating the downstream task of object detection.

Key Insights and Ideas

  • Latent space supervision is key: The paper highlights that simply enforcing cycle-consistency and adversarial losses in CycleGAN isn't enough for high-quality restoration, especially with noisy intermediate outputs. Introducing supervision in the latent space, where the representations are more abstract, proves highly effective.
  • Deep GPs for uncertainty-aware pseudo-labels: Leveraging Deep GPs allows not only for generating pseudo-labels but also for quantifying the uncertainty associated with these labels. This uncertainty-aware approach is critical for filtering out noisy predictions and improving training stability.
  • Weather-agnostic restoration: Unlike many prior unsupervised methods that rely on specific weather characteristics, their proposed DGP-CycleGAN is designed to be general and can be applied to various weather conditions without modification.
  • Bridging the synthetic-to-real gap: By enabling training on real-world, unpaired data, the method directly addresses the domain shift problem that plagues methods trained solely on synthetic data.

Key Results

  • De-raining: The proposed DGP-CycleGAN consistently outperforms CycleGAN and other unsupervised methods, achieving significant improvements in PSNR and SSIM. For instance, it shows a minimum improvement of 1.43 dB in PSNR on Rain100H and achieves results comparable to supervised SOTA methods like SPANet and PReNet.
  • De-hazing: Similar to de-raining, DGP-CycleGAN shows significant improvements over CycleGAN and other unsupervised techniques, outperforming them by a large margin while remaining competitive with fully-supervised SOTA methods.
  • De-snowing: The method achieves results comparable to oracle (fully-supervised upper bound) and SOTA supervised techniques, while being in the unsupervised category.
  • Downstream task performance: When evaluated on object detection, the restored images from DGP-CycleGAN lead to a higher overall mAP (34.3) compared to CycleGAN (33.7) and feature-level adaptation methods, demonstrating the improved quality of the restored images.
  • Generalization: The proposed loss functions can also be applied to other unpaired translation frameworks like UNIT GAN, showing improved performance.

Limitations and Future Work

  • Computational cost: While not explicitly stated as a limitation, the use of GPs can increase computational complexity, especially with larger datasets or deeper networks.
  • Ablation studies: The paper includes extensive ablation studies on hyperparameters like the number of GP layers (L), nearest neighbors (Nn), and the loss weight (λp), suggesting that these factors are important for optimal performance.
  • Future work: The authors suggest that their approach can be extended to other unpaired translation tasks beyond weather restoration. They also demonstrate the effectiveness of their method on a downstream task (object detection), hinting at potential applications in improving the robustness of other vision systems.

Summary updated

Abstract

Existing approaches for restoring weather-degraded images follow a fully-supervised paradigm and they require paired data for training. However, collecting paired data for weather degradations is extremely challenging, and existing methods end up training on synthetic data. To overcome this issue, we describe an approach for supervising deep networks that are based on CycleGAN, thereby enabling the use of unlabeled real-world data for training. Specifically, we introduce new losses for training CycleGAN that lead to more effective training, resulting in high-quality reconstructions. These new losses are obtained by jointly modeling the latent space embeddings of predicted clean images and original clean images through Deep Gaussian Processes. This enables the CycleGAN architecture to transfer the knowledge from one domain (weather-degraded) to another (clean) more effectively. We demonstrate that the proposed method can be effectively applied to different restoration tasks like de-raining, de-hazing and de-snowing and it outperforms other unsupervised techniques (that leverage weather-based characteristics) by a considerable margin.
BibTeX
@misc{yasarla2022unsupervised,
  author = {Rajeev Yasarla and Vishwanath A. Sindagi and Vishal M. Patel},
  title = {{Unsupervised Restoration of Weather-affected Images using Deep Gaussian Process-based CycleGAN}},
  year = {2022},
  eprint = {2204.10970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.10970},
}
Figure