An Efficient Dehazing Algorithm Based on the Fusion of Transformer and Convolutional Neural Network

Jun Xu, Zixuan Chen, Hao Luo, Zhe‐Ming LuSensors 2022 · DOI

TL;DR

This paper introduces a novel Transformer-Convolution Fusion Dehazing Network (TCFDN) that effectively combines the global context-aware capabilities of Transformers with the local feature extraction strengths of Convolutional Neural Networks (CNNs). By adaptively fusing these two powerful approaches, TCFDN achieves state-of-the-art performance in single image dehazing, producing clearer and more natural-looking results than existing methods.

What Problem Does This Paper Address?

Images captured in hazy or foggy conditions suffer from reduced visibility, color distortion, and loss of detail. This degradation significantly impacts the performance of various computer vision tasks, such as object detection, surveillance, and autonomous driving. While image dehazing aims to restore clarity, traditional methods often struggle with complex atmospheric conditions or are computationally expensive. Deep learning has shown great promise, but existing approaches often excel at either global context or local details, but not both simultaneously. This paper tackles the challenge of creating a single-image dehazing algorithm that can effectively leverage both global and local image information for superior restoration.

Methods

The core innovation of this paper is the Transformer-Convolution Fusion Dehazing Network (TCFDN), built upon a classic self-encoder architecture. The key novelty lies in the Transformer-Convolution hybrid layer. This layer intelligently integrates:

  • Swin-Transformer: Leverages its ability to capture long-range dependencies and global context within an image.
  • Convolutional Neural Network (CNN) blocks: Utilizes their strength in extracting local features and fine-grained details.

Crucially, these two components are fused using an adaptive fusion mechanism. This mechanism learns to dynamically weigh the contributions of the Transformer and CNN outputs, allowing the network to prioritize the most relevant information for dehazing at different stages. The paper also highlights the use of specific convolutional blocks (FA Block and deformable convolution) that are particularly well-suited for image dehazing tasks, enhancing feature extraction and reconstruction.

Datasets

The paper primarily uses the RESIDE dataset for training and evaluation. Specifically:

  • ITS (Indoor Training Set): Used for training the model.
  • SOTS (Synthetic Objective Testing Set) - Indoor: Used as the primary test set for objective performance evaluation.

The authors also mention using real-world hazy images for subjective evaluation.

Key Insights and Ideas

  • Synergy of Global and Local: The paper strongly demonstrates that combining Transformer's global receptive field with CNN's local inductive bias is more effective for image dehazing than using either alone.
  • Adaptive Fusion is Key: The adaptive fusion mechanism is not just a simple concatenation; it learns to intelligently blend information, suggesting that the optimal balance between global and local features can vary and should be learned.
  • Specialized CNNs Matter: The choice of specific CNN blocks (like FA Block and deformable convolution) tailored for dehazing tasks can significantly boost performance, indicating that architectural choices within the CNN component are important.
  • Transformer's Potential Beyond Vision Tasks: This work further solidifies the Transformer's applicability to low-level vision tasks like image restoration, moving beyond its initial success in classification and detection.

Key Results

The TCFDN achieved state-of-the-art results on the SOTS indoor dataset, outperforming several prominent deep learning-based dehazing algorithms.

  • Objective Performance:
    • Achieved a PSNR of 37.62 and an SSIM of 0.9910 on the SOTS indoor test set.
  • This surpassed the previous best-performing method, AECR-Net, by 0.5dB in PSNR and 0.001 in SSIM.
  • Subjective Performance:
    • Qualitative results on both synthetic and real-world hazy images show that TCFDN produces visually superior results, with better detail preservation, more natural colors, and fewer artifacts compared to traditional methods (DCP) and other advanced deep learning models (MSBDN, FFA-Net).
  • The network demonstrated good generalization ability, effectively dehazing real-world images even though it was primarily trained on synthetic data.
  • Ablation Studies:
    • Experiments replacing the hybrid layer with only Swin-Transformer or only CNN blocks showed significantly lower performance (PSNR of 34.68 and 35.65, respectively), underscoring the effectiveness of the proposed fusion approach.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the Swin-Transformer reduces some limitations of standard Transformers, it still has limitations in modeling very fine local details compared to CNNs.
  • The reliance on synthetic datasets for training, while common, might not perfectly capture the nuances of all real-world haze conditions.
  • Future Work:
    • The authors suggest that the Transformer-Convolution hybrid layer's structure can be further optimized.
  • Exploring different adaptive fusion strategies and more advanced convolutional blocks could lead to further improvements.
  • Investigating unsupervised or semi-supervised learning approaches for dehazing could reduce the reliance on large labeled datasets.

Summary updated

Abstract

The purpose of image dehazing is to remove the interference from weather factors in degraded images and enhance the clarity and color saturation of images to maximize the restoration of useful features. Single image dehazing is one of the most important tasks in the field of image restoration. In recent years, due to the progress of deep learning, single image dehazing has made great progress. With the success of Transformer in advanced computer vision tasks, some research studies also began to apply Transformer to image dehazing tasks and obtained surprising results. However, both the deconvolution-neural-network-based dehazing algorithm and Transformer based dehazing algorithm magnify their advantages and disadvantages separately. Therefore, this paper proposes a novel Transformer-Convolution fusion dehazing network (TCFDN), which uses Transformer's global modeling ability and convolutional neural network's local modeling ability to improve the dehazing ability. In the Transformer-Convolution fusion dehazing network, the classic self-encoder structure is used. This paper proposes a Transformer-Convolution hybrid layer, which uses an adaptive fusion strategy to make full use of the Swin-Transformer and convolutional neural network to extract and reconstruct image features. On the basis of previous research, this layer further improves the ability of the network to remove haze. A series of contrast experiments and ablation experiments not only proved that the Transformer-Convolution fusion dehazing network proposed in this paper exceeded the more advanced dehazing algorithm, but also provided solid and powerful evidence for the basic theory on which it depends.
BibTeX
@inproceedings{xu2022efficient,
  author = {Jun Xu and Zixuan Chen and Hao Luo and Zhe‐Ming Lu},
  title = {{An Efficient Dehazing Algorithm Based on the Fusion of Transformer and Convolutional Neural Network}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s23010043},
}
Figure