NightHazeFormer: Single Nighttime Haze Removal Using Prior Query Transformer

Yun Liu, Zhongsheng Yan, Sixiang Chen, Tian Ye, Wenqi Ren, Erkang Chen

Introduction

Nighttime dehazing is a significantly more complex task than daytime dehazing due to the presence of multiple artificial light sources. While daytime models typically assume a single global light source, nighttime scenes are characterized by localized glow, extreme shadows, and uneven illumination. NightHazeFormer addresses these challenges through a novel two-stage transformer framework that integrates physical priors and semi-supervised learning to bridge the gap between synthetic training and real-world application.

Methodology

The NightHazeFormer framework operates in two distinct phases to handle the unique degradations of nighttime photography.

Supervised Pre-training with Prior Queries

The core architecture is an encoder-decoder transformer utilizing NAFBlocks (Non-Aligned Feature blocks). A key innovation is the Prior Query Transformer in the decoder, which injects classical physical priors—the Dark Channel Prior (DCP) and the Bright Channel Prior (BCP)—as explicit queries. These priors act as guides for the self-attention mechanism, helping the model focus on hazy regions (via DCP) while maintaining contrast and brightness (via BCP).

Semi-supervised Fine-tuning

To adapt to real-world domain shifts, the model undergoes a semi-supervised refinement stage. Using unlabeled real-world hazy images, the system generates "pseudo ground truths" via an unsupervised dehazing method (BCCR). These pseudo-labels facilitate a cyclic supervised fine-tuning process, allowing the network to learn the specific distribution of real-world nighttime degradations without requiring expensive, manually labeled real-world pairs.

Datasets: Simulating Real-World Complexity

The authors introduced the UNREAL-NH dataset, a large-scale synthetic collection of 10,080 image pairs created using Unreal Engine 4.27. This dataset simulates complex optical effects such as bloom, motion blur, and lens flares that are often ignored in simpler mathematical haze models.

Statistical analysis of dataset realism. The t-SNE map (i) and FID scores (j) demonstrate that the UNREAL-NH dataset (red) aligns more closely with real-world distributions (blue) than previous synthetic benchmarks.

Statistical evaluations using t-SNE maps and Fréchet Inception Distance (FID) confirm that UNREAL-NH is significantly more representative of real-world data than existing benchmarks like NHC or GTA5. For instance, UNREAL-NH achieved an FID of 12.94, the lowest among all synthetic sets compared.

Experimental Results

NightHazeFormer demonstrates superior performance in both quantitative metrics and visual quality across various benchmarks.

Comprehensive comparison of dehazing results. Panels (a-h) show visual outputs from various methods on a sample scene, while (i-j) provide statistical context for the datasets used.

Quantitative Performance

On the UNREAL-NH benchmark, NightHazeFormer achieved a PSNR of 27.92 dB, a substantial improvement over the previous state-of-the-art learning-based method, CFEN-ViT, which scored 19.17 dB.

Qualitative Analysis

Visually, the model effectively suppresses the "halo" or glow effects around streetlights, a common failure point for other methods. It restores sharp details in dark areas without introducing the color shifts or excessive darkening seen in earlier approaches. The model also performed best on real-world images (REAL-NH) according to non-reference metrics like NIQE and MUSIQ.

Conclusion

NightHazeFormer represents a significant advancement in nighttime image restoration. By leveraging physical priors as transformer queries and utilizing a semi-supervised adaptation strategy, the model successfully navigates the complexities of artificial lighting and haze. The accompanying UNREAL-NH dataset provides a robust foundation for future developments in the field, offering a more realistic benchmark for synthetic-to-real generalization.

Summary updated

Abstract

Nighttime image dehazing is a challenging task due to the presence of multiple types of adverse degrading effects including glow, haze, blur, noise, color distortion, and so on. However, most previous studies mainly focus on daytime image dehazing or partial degradations presented in nighttime hazy scenes, which may lead to unsatisfactory restoration results. In this paper, we propose an end-to-end transformer-based framework for nighttime haze removal, called NightHazeFormer. Our proposed approach consists of two stages: supervised pre-training and semi-supervised fine-tuning. During the pre-training stage, we introduce two powerful priors into the transformer decoder to generate the non-learnable prior queries, which guide the model to extract specific degradations. For the fine-tuning, we combine the generated pseudo ground truths with input real-world nighttime hazy images as paired images and feed into the synthetic domain to fine-tune the pre-trained model. This semi-supervised fine-tuning paradigm helps improve the generalization to real domain. In addition, we also propose a large-scale synthetic dataset called UNREAL-NH, to simulate the real-world nighttime haze scenarios comprehensively. Extensive experiments on several synthetic and real-world datasets demonstrate the superiority of our NightHazeFormer over state-of-the-art nighttime haze removal methods in terms of both visually and quantitatively.
BibTeX
@inproceedings{liu2023nighthazeformer,
  author = {Yun Liu and Zhongsheng Yan and Sixiang Chen and Tian Ye and Wenqi Ren and Erkang Chen},
  title = {{NightHazeFormer: Single Nighttime Haze Removal Using Prior Query Transformer}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611744},
}
Figure