Abstract
Most dehazing methods suffer from limited receptive field and do not explore the rich semantic prior encapsulated in vision-language models, which have proven effective in downstream tasks. In this paper, we introduce CLIPHaze, a pioneering hybrid framework that synergizes the efficient global modeling of Mamba with the prior knowledge and zero-shot capabilities of CLIP to address both issues simultaneously. Specifically, our method employs parallel state space model and window-based self-attention to obtain global contextual dependency and local fine-grained perception, respectively. To seamlessly aggregate information from both paths, we introduce CLIP-instructed Aggregation Module (CAM). For non-homogeneous and homogeneous haze, CAM leverages zero-shot estimated haze density map and high-quality image embedding without degradation information to explicitly and implicitly determine the optimal neural operation range for each pixel, thereby adaptively fusing two paths with different receptive fields. Extensive experiments on various benchmarks demonstrate that CLIPHaze achieves state-of-the-art (SOTA) performance, particularly in non-homogeneous haze.
Introduction
Image dehazing is a classic ill-posed problem in computer vision, requiring the reconstruction of a clear scene from light scattered and absorbed by atmospheric particles. Traditional deep learning approaches typically prioritize either local textures or global context. Local-focused methods, utilizing convolutions or window-based attention, excel at preserving fine details but often struggle with heavy, non-homogeneous haze. Conversely, global-focused methods, such as State Space Models (Mamba) or linear attention, provide necessary long-range dependencies but can blur local 2D structures.
The core challenge lies in the fact that not every pixel requires the same degree of contextual information. A patch obscured by thick haze necessitates broader global information to infer underlying structures, while clearer regions benefit more from local refinement. CLIPHaze (also referred to as DehazeMatic) addresses this by dynamically balancing these two ranges through a novel aggregation strategy.
Methods
The CLIPHaze framework introduces two primary architectural innovations to optimize the trade-off between local and global perception.
TrambaDA Block: Dual-Path Perception
The Transformer-Mamba Dual Aggregation (TrambaDA) block processes features through two parallel streams:
- Short-range Path: Employs window-based self-attention to maintain high-fidelity local textures and fine-grained perception.
- Long-range Path: Utilizes a four-directional Mamba (S6) block to efficiently capture global contextual dependencies across the entire image.
CedA Module: CLIP-Enhanced Aggregation
The CLIP-enhanced Dual-path Aggregator (CedA), or CLIP-instructed Aggregation Module (CAM), acts as the system's "instructor." It leverages a frozen CLIP image encoder to guide the fusion of the dual paths.
The aggregation is driven by two key mechanisms:
- Haze Density Mapping: By utilizing learnable text prompts (e.g., "hazy" vs. "clear"), the model calculates similarity between CLIP's latent space and image patches. This enables the generation of fine-grained haze density maps without requiring explicit density labels during training.
- Semantic Refinement: CLIP’s rich semantic priors are integrated via bidirectional cross-attention, aligning high-level semantic features with low-level architectural features to ensure content-aware restoration.
Experimental Results
CLIPHaze was evaluated against a wide range of benchmarks, including synthetic homogeneous haze (RESIDE/SOTS), synthetic non-homogeneous haze (NH-HAZE, Dense-Haze), and real-world datasets (RTTS).
Performance and Efficiency
The model achieves state-of-the-art results across major metrics. On the SOTS average benchmark, it reached a PSNR of 39.86 dB, significantly surpassing previous leaders like DCMPNet. Its performance is particularly notable in non-homogeneous conditions, achieving 21.47 dB on NH-HAZE.
Despite the inclusion of a CLIP backbone, the architecture is highly optimized. The CLIP inference adds only ~3.8ms of latency, allowing the full model to run at 33 FPS on an NVIDIA A100. This makes CLIPHaze suitable for real-time video dehazing applications.
Conclusion
CLIPHaze demonstrates that vision-language models like CLIP can serve as effective "degradation sensors" for low-level vision tasks. By using CLIP to intelligently weight local and global dependencies, the model achieves superior restoration quality, especially in complex, non-homogeneous haze. Future work may extend this "aggregation instructor" concept to other restoration tasks such as desnowing or low-light enhancement, potentially utilizing more nuanced weather-aware prompts.
Summary updated