TL;DR
This paper introduces PSD, a novel framework that bridges the gap between synthetic and real-world image dehazing. By pre-training on synthetic data and then fine-tuning with unsupervised learning guided by physical priors, PSD significantly improves the generalization of dehazing models to real-world scenarios, achieving state-of-the-art performance.
What Problem Does This Paper Address?
Deep learning models have shown impressive results in image dehazing, but they typically rely on large datasets of paired synthetic hazy and clean images for training. A major challenge arises when these models are applied to real-world hazy images, as there's often a significant performance drop due to the "domain gap" between synthetic and real data. This is because real-world haze formation and image capture have complexities not fully captured by synthetic datasets. Obtaining real-world paired hazy and clean images is extremely difficult, making it hard to train models that generalize well. This paper tackles this problem by proposing a principled framework to adapt models trained on synthetic data to perform effectively on real-world hazy images without requiring real-world ground truth.
Methods
The proposed framework, Principled Synthetic-to-real Dehazing (PSD), operates in two main stages:
- Supervised Pre-training: A chosen dehazing model backbone (e.g., MSBDN) is modified to be physics-based. This involves adding modules to estimate not just the clean image but also the transmission map and atmospheric light. The model is then trained on synthetic hazy/clean image pairs using standard reconstruction losses. This stage leverages existing strong dehazing models and imbues them with physical understanding.
- Unsupervised Fine-tuning: The pre-trained model is then fine-tuned using unlabeled real-world hazy images. The key innovation here is the use of a "prior loss committee." This committee comprises several well-established physical priors (Dark Channel Prior, Bright Channel Prior, and CLAHE reconstruction loss) that act as a proxy for ground truth. These priors guide the model to learn the characteristics of real-world haze-free images. Additionally, a "learning without forgetting" (LWF) loss is used to retain knowledge from the synthetic domain, and a "sky loss" is incorporated to handle artifacts in sky regions. The overall fine-tuning loss combines these prior losses with the reconstruction loss and LWF loss.
Datasets
- Training:
- OTS (Outdoor Training Set): Used for pre-training the model on synthetic hazy/clean image pairs.
- URHI (Unannotated Real Hazy Images): Used for unsupervised fine-tuning on real-world hazy images.
- Evaluation:
- RTTS (Real-world Testing Set): A subset of the RESIDE dataset used for quantitative and task-driven evaluations.
- HSTS: A subset of RESIDE used for human subjective evaluation.
- Real hazy images from authors of previous studies: Used for qualitative visual comparisons.
Key Insights and Ideas
- Bridging the Domain Gap with Physical Priors: The core idea is that even without ground truth, real-world clean images adhere to certain physical rules. By formulating these rules as loss functions (the "prior loss committee"), the model can be guided to learn these underlying principles, effectively adapting to the real domain.
- Complementarity of Priors: The paper emphasizes that no single physical prior is perfect. Combining multiple priors (DCP, BCP, CLAHE) leads to more robust and stable fine-tuning, with each prior contributing to different aspects of image quality and compensating for the weaknesses of others.
- Plug-and-Play Framework: PSD is designed to be versatile, allowing most existing deep dehazing models to be used as backbones, making it a broadly applicable solution.
- Unsupervised Learning for Real-World Data: The framework effectively leverages unlabeled real-world data, which is abundant and more representative of actual deployment scenarios, overcoming the limitations of synthetic-only training.
Key Results
- State-of-the-Art Performance: PSD achieves state-of-the-art results on real-world dehazing tasks, outperforming existing methods in visual quality, no-reference image quality assessment (NIQE, BRISQUE, NIMA), and downstream task performance (object detection using YOLOv3).
- Visual Quality: Qualitative results show PSD generating cleaner, brighter images with sharper details and fewer artifacts compared to other methods.
- Quantitative Metrics: PSD achieved the best scores on three out of four no-reference image quality assessment metrics (BRISQUE, NIQE, NIMA) on the RTTS dataset.
- Downstream Task Performance: Object detection (mAP) improved by 2.52% when using PSD as a pre-processing step compared to the hazy input, demonstrating its effectiveness in improving real-world vision applications.
- Ablation Studies: The ablation study confirmed the importance of each prior in the loss committee, with the full committee yielding the best results.
Limitations and Future Work
- Limitations:
- The paper notes that physical priors can sometimes struggle with sky regions, leading to artifacts. While they introduce a "sky loss" to mitigate this, it's an area that could be further improved.
- The performance of FADE (a haze density metric) was slightly lower for PSD compared to SSLD, as FADE primarily focuses on haze density and not necessarily overall image detail and color fidelity, where PSD excels.
- Future Work:
- The authors suggest that the PSD framework can be extended to other image restoration tasks that suffer from similar synthetic-to-real domain gaps.
- Further exploration of more sophisticated physical priors or novel ways to combine them could lead to even better performance.
Summary updated