DEHRFormer: Real-Time Transformer for Depth Estimation and Haze Removal from Varicolored Haze Scenes

Tian Ye, Sixiang Chen, Tian Ye 0001, Jun Shi, Yun Liu, Jingxia Jiang, Erkang Chen, Peng Chen

TL;DR

DEHRFormer is a novel, real-time transformer architecture designed to simultaneously remove haze and estimate scene depth, specifically targeting the challenge of "varicolored" haze (haze with distinct color casts). By unifying these tasks and employing a semi-supervised contrastive learning strategy, it achieves state-of-the-art performance and robust real-world generalization at over 30 FPS.

What Problem Does This Paper Address?

In computer vision, haze removal (dehazing) and depth estimation are traditionally treated as separate, sequential problems. Most pipelines dehaze an image first and then pass the result to a depth estimation model. However, this approach has two major flaws:

  1. Lost Connections: The physics of haze is intrinsically tied to depth—the further away an object is, the denser the haze. Sequential processing loses these "inner connections."
  2. The Color Cast Challenge: Most research assumes haze is a uniform gray. In reality, haze often has chromatic casts (varicolored haze) due to lighting and atmospheric particles. Standard models often fail to remove these color shifts or produce distorted depth maps when faced with them.

Furthermore, models trained on synthetic data often suffer from a "domain gap," failing to generalize to the messy, unpredictable nature of real-world hazy scenes.

Methods

The authors introduce DEHRFormer, a unified transformer-based framework consisting of a single encoder and two task-specific decoders.

  • Coupling Learning Encoder: Unlike standard transformers that use computationally expensive self-attention in the encoder ($O(N^2)$), DEHRFormer uses a CNN-based encoder with Multi-Scale Feature Modeling (MSFM) blocks. This maintains $O(N)$ efficiency, enabling real-time performance.
  • Task-Specific Decoders with Learnable Queries: Inspired by DETR, the decoders use learnable queries to "probe" the encoder's features.
  • The Dehazing Decoder uses "haze type queries" to adaptively learn and remove diverse color degradations.
  • The Depth Decoder is serial to the dehazing process, leveraging cleaned features to ensure depth is estimated from a "haze-free" perspective while maintaining the structural context of the original scene.
  • Semi-supervised Contrastive Learning: To bridge the domain gap, the authors use real-world hazy images as "negative samples" in a contrastive learning framework. This teaches the model to push its predictions away from "hazy" states and toward "clean" states, even without ground-truth labels for the real-world data.
  • Domain Consistency Learning: A specific loss function ensures that the model predicts the same depth map for a scene, regardless of whether it is looking at the hazy version or the clean version.

Datasets

The study utilizes several key datasets:

  • VHRDE (Varicolored Haze Scene Depth Estimation): A new dataset introduced by the authors containing 8,000 paired images (6,000 for training, 2,000 for testing) specifically featuring varicolored haze and corresponding depth maps.
  • URHI (Unannotated Real Hazy Images): 2,000 real-world hazy images used for the semi-supervised contrastive learning phase.
  • NIMA: Used for no-reference image quality assessment on real-world results.

Key Insights and Ideas

  • Haze as a Feature, Not Just Noise: Instead of seeing haze as something to be discarded before depth estimation, DEHRFormer treats the degradation itself as a depth cue, preserving the mathematical relationship defined by the atmospheric scattering model.
  • Real-World Negatives: One of the "aha" moments is the use of unlabelled real-world hazy images as negative samples. This allows the model to learn the "texture" of real haze without needing expensive pixel-perfect labels, significantly boosting its ability to handle real-world photos.
  • Efficiency through Hybridization: By combining a CNN encoder (for speed) with transformer decoders (for global context and adaptive querying), the model achieves a "best of both worlds" result: high accuracy and real-time inference.

Key Results

DEHRFormer outperforms existing state-of-the-art (SOTA) methods like NAFNet and PMNet across both tasks:

  • Dehazing: Achieved a PSNR of 23.42 dB and SSIM of 0.876 on the VHRDE dataset, surpassing the previous best (NAFNet) by 0.41 dB.
  • Depth Estimation: Achieved an RMSE of 0.286, significantly better than sequential pipelines (e.g., SDDE at 0.299).
  • Speed: The model runs in 0.034 seconds (approx. 29.4 FPS) on an NVIDIA RTX 3090, making it suitable for real-time video processing.
  • Qualitative: Visual results show much cleaner removal of color casts (like blue or yellow tints) compared to traditional methods, which often leave "residual haze" in the background.

Limitations and Future Work

While DEHRFormer excels at varicolored haze, the authors note that the performance is sensitive to the number of learnable queries; there is a "sweet spot" (around 64 queries) beyond which memory costs increase without significant gains.

Future work could explore more complex ratios of positive to negative samples in the contrastive learning stage. Additionally, while the model handles color casts well, testing it against extreme "non-haze" degradations (like heavy rain or night-time glare) remains an open area for expansion.

Summary updated

Abstract

Varicolored haze caused by chromatic casts poses haze removal and depth estimation challenges. Recent learning-based depth estimation methods are mainly targeted at dehazing first and estimating depth subsequently from haze-free scenes. This way, the inner connections between colored haze and scene depth are lost. In this paper, we propose a real-time transformer for simultaneous single image Depth Estimation and Haze Removal (DEHRFormer). DEHRFormer consists of a single encoder and two task-specific decoders. The transformer decoders with learnable queries are designed to decode coupling features from the task-agnostic encoder and project them into clean image and depth map, respectively. In addition, we introduce a novel learning paradigm that utilizes contrastive learning and domain consistency learning to tackle weak-generalization problem for real-world dehazing, while predicting the same depth map from the same scene with varicolored haze. Experiments demonstrate that DEHRFormer achieves significant performance improvement across diverse varicolored haze scenes over previous depth estimation networks and dehazing approaches.
BibTeX
@inproceedings{ye2023dehrformer,
  author = {Tian Ye and Sixiang Chen and Tian Ye and Jun Shi and Yun Liu and Jingxia Jiang and Erkang Chen and Peng Chen},
  title = {{DEHRFormer: Real-Time Transformer for Depth Estimation and Haze Removal from Varicolored Haze Scenes}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096828},
}

Loading related content...

Figure