NightHazeFormer: Single Nighttime Haze Removal Using Prior Query Transformer

Yun Liu, Zhongsheng Yan, Sixiang Chen, Tian Ye, Wenqi Ren, Erkang Chen

Introduction

Nighttime dehazing is a significantly more complex task than daytime dehazing due to the presence of multiple artificial light sources. While daytime models typically assume a single global light source, nighttime scenes are characterized by localized glow, extreme shadows, and uneven illumination. NightHazeFormer addresses these challenges through a novel two-stage transformer framework that integrates physical priors and semi-supervised learning to bridge the gap between synthetic training and real-world application.

Methodology

The NightHazeFormer framework operates in two distinct phases to handle the unique degradations of nighttime photography.

Supervised Pre-training with Prior Queries

The core architecture is an encoder-decoder transformer utilizing NAFBlocks (Non-Aligned Feature blocks). A key innovation is the Prior Query Transformer in the decoder, which injects classical physical priors—the Dark Channel Prior (DCP) and the Bright Channel Prior (BCP)—as explicit queries. These priors act as guides for the self-attention mechanism, helping the model focus on hazy regions (via DCP) while maintaining contrast and brightness (via BCP).

Semi-supervised Fine-tuning

To adapt to real-world domain shifts, the model undergoes a semi-supervised refinement stage. Using unlabeled real-world hazy images, the system generates "pseudo ground truths" via an unsupervised dehazing method (BCCR). These pseudo-labels facilitate a cyclic supervised fine-tuning process, allowing the network to learn the specific distribution of real-world nighttime degradations without requiring expensive, manually labeled real-world pairs.

Datasets: Simulating Real-World Complexity

The authors introduced the UNREAL-NH dataset, a large-scale synthetic collection of 10,080 image pairs created using Unreal Engine 4.27. This dataset simulates complex optical effects such as bloom, motion blur, and lens flares that are often ignored in simpler mathematical haze models.

Statistical analysis of dataset realism. The t-SNE map (i) and FID scores (j) demonstrate that the UNREAL-NH dataset (red) aligns more closely with real-world distributions (blue) than previous synthetic benchmarks.

Statistical evaluations using t-SNE maps and Fréchet Inception Distance (FID) confirm that UNREAL-NH is significantly more representative of real-world data than existing benchmarks like NHC or GTA5. For instance, UNREAL-NH achieved an FID of 12.94, the lowest among all synthetic sets compared.

Experimental Results

NightHazeFormer demonstrates superior performance in both quantitative metrics and visual quality across various benchmarks.

Comprehensive comparison of dehazing results. Panels (a-h) show visual outputs from various methods on a sample scene, while (i-j) provide statistical context for the datasets used.

Quantitative Performance

On the UNREAL-NH benchmark, NightHazeFormer achieved a PSNR of 27.92 dB, a substantial improvement over the previous state-of-the-art learning-based method, CFEN-ViT, which scored 19.17 dB.

Qualitative Analysis

Visually, the model effectively suppresses the "halo" or glow effects around streetlights, a common failure point for other methods. It restores sharp details in dark areas without introducing the color shifts or excessive darkening seen in earlier approaches. The model also performed best on real-world images (REAL-NH) according to non-reference metrics like NIQE and MUSIQ.

Conclusion

NightHazeFormer represents a significant advancement in nighttime image restoration. By leveraging physical priors as transformer queries and utilizing a semi-supervised adaptation strategy, the model successfully navigates the complexities of artificial lighting and haze. The accompanying UNREAL-NH dataset provides a robust foundation for future developments in the field, offering a more realistic benchmark for synthetic-to-real generalization.

Summary updated

BibTeX
@inproceedings{liu2023nighthazeformer,
  author = {Yun Liu and Zhongsheng Yan and Sixiang Chen and Tian Ye and Wenqi Ren and Erkang Chen},
  title = {{NightHazeFormer: Single Nighttime Haze Removal Using Prior Query Transformer}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3611744},
}

Loading related content...

Figure