PSD: Principled Synthetic-to-Real Dehazing Guided by Physical Priors

Zeyuan Chen, Yangchao Wang, Yang Yang, Dong LiuCVPR 2021 · DOI · OpenReview

TL;DR

This paper introduces PSD, a novel framework that bridges the gap between synthetic and real-world image dehazing. By pre-training on synthetic data and then fine-tuning with unsupervised learning guided by physical priors, PSD significantly improves the generalization of dehazing models to real-world scenarios, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Deep learning models have shown impressive results in image dehazing, but they typically rely on large datasets of paired synthetic hazy and clean images for training. A major challenge arises when these models are applied to real-world hazy images, as there's often a significant performance drop due to the "domain gap" between synthetic and real data. This is because real-world haze formation and image capture have complexities not fully captured by synthetic datasets. Obtaining real-world paired hazy and clean images is extremely difficult, making it hard to train models that generalize well. This paper tackles this problem by proposing a principled framework to adapt models trained on synthetic data to perform effectively on real-world hazy images without requiring real-world ground truth.

Methods

The proposed framework, Principled Synthetic-to-real Dehazing (PSD), operates in two main stages:

  1. Supervised Pre-training: A chosen dehazing model backbone (e.g., MSBDN) is modified to be physics-based. This involves adding modules to estimate not just the clean image but also the transmission map and atmospheric light. The model is then trained on synthetic hazy/clean image pairs using standard reconstruction losses. This stage leverages existing strong dehazing models and imbues them with physical understanding.
  2. Unsupervised Fine-tuning: The pre-trained model is then fine-tuned using unlabeled real-world hazy images. The key innovation here is the use of a "prior loss committee." This committee comprises several well-established physical priors (Dark Channel Prior, Bright Channel Prior, and CLAHE reconstruction loss) that act as a proxy for ground truth. These priors guide the model to learn the characteristics of real-world haze-free images. Additionally, a "learning without forgetting" (LWF) loss is used to retain knowledge from the synthetic domain, and a "sky loss" is incorporated to handle artifacts in sky regions. The overall fine-tuning loss combines these prior losses with the reconstruction loss and LWF loss.

Datasets

  • Training:
    • OTS (Outdoor Training Set): Used for pre-training the model on synthetic hazy/clean image pairs.
  • URHI (Unannotated Real Hazy Images): Used for unsupervised fine-tuning on real-world hazy images.
  • Evaluation:
    • RTTS (Real-world Testing Set): A subset of the RESIDE dataset used for quantitative and task-driven evaluations.
  • HSTS: A subset of RESIDE used for human subjective evaluation.
  • Real hazy images from authors of previous studies: Used for qualitative visual comparisons.

Key Insights and Ideas

  • Bridging the Domain Gap with Physical Priors: The core idea is that even without ground truth, real-world clean images adhere to certain physical rules. By formulating these rules as loss functions (the "prior loss committee"), the model can be guided to learn these underlying principles, effectively adapting to the real domain.
  • Complementarity of Priors: The paper emphasizes that no single physical prior is perfect. Combining multiple priors (DCP, BCP, CLAHE) leads to more robust and stable fine-tuning, with each prior contributing to different aspects of image quality and compensating for the weaknesses of others.
  • Plug-and-Play Framework: PSD is designed to be versatile, allowing most existing deep dehazing models to be used as backbones, making it a broadly applicable solution.
  • Unsupervised Learning for Real-World Data: The framework effectively leverages unlabeled real-world data, which is abundant and more representative of actual deployment scenarios, overcoming the limitations of synthetic-only training.

Key Results

  • State-of-the-Art Performance: PSD achieves state-of-the-art results on real-world dehazing tasks, outperforming existing methods in visual quality, no-reference image quality assessment (NIQE, BRISQUE, NIMA), and downstream task performance (object detection using YOLOv3).
  • Visual Quality: Qualitative results show PSD generating cleaner, brighter images with sharper details and fewer artifacts compared to other methods.
  • Quantitative Metrics: PSD achieved the best scores on three out of four no-reference image quality assessment metrics (BRISQUE, NIQE, NIMA) on the RTTS dataset.
  • Downstream Task Performance: Object detection (mAP) improved by 2.52% when using PSD as a pre-processing step compared to the hazy input, demonstrating its effectiveness in improving real-world vision applications.
  • Ablation Studies: The ablation study confirmed the importance of each prior in the loss committee, with the full committee yielding the best results.

Limitations and Future Work

  • Limitations:
    • The paper notes that physical priors can sometimes struggle with sky regions, leading to artifacts. While they introduce a "sky loss" to mitigate this, it's an area that could be further improved.
  • The performance of FADE (a haze density metric) was slightly lower for PSD compared to SSLD, as FADE primarily focuses on haze density and not necessarily overall image detail and color fidelity, where PSD excels.
  • Future Work:
    • The authors suggest that the PSD framework can be extended to other image restoration tasks that suffer from similar synthetic-to-real domain gaps.
  • Further exploration of more sophisticated physical priors or novel ways to combine them could lead to even better performance.

Summary updated

Abstract

Deep learning-based methods have achieved remarkable performance for image dehazing. However, previous studies are mostly focused on training models with synthetic hazy images, which incurs performance drop when the models are used for real-world hazy images. We propose a Principled Synthetic-to-real Dehazing (PSD) framework to improve the generalization performance of dehazing. Starting from a dehazing model backbone that is pre-trained on synthetic data, PSD exploits real hazy images to fine-tune the model in an unsupervised fashion. For the fine-tuning, we leverage several well-grounded physical priors and combine them into a prior loss committee. PSD allows for most of the existing dehazing models as its backbone, and the combination of multiple physical priors boosts dehazing significantly. Through extensive experiments, we demonstrate that our PSD framework establishes the new state-of-the-art performance for real-world dehazing, in terms of visual quality assessed by no-reference quality metrics as well as subjective evaluation and downstream task performance indicator.
BibTeX
@inproceedings{chen2021psd,
  author = {Zeyuan Chen and Yangchao Wang and Yang Yang and Dong Liu},
  title = {{PSD: Principled Synthetic-to-Real Dehazing Guided by Physical Priors}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00710},
}
Figure