Multi-Branch Gated Fusion Network: A Method That Provides Higher-Quality Images for the USV Perception System in Maritime Hazy Condition

Yunsheng Fan, Longhui Niu, Ting LiuJournal of Marine Science and Engineering 2022 · DOI

TL;DR

This paper introduces the Multi-Branch Gated Fusion Network (MGFNet), a novel deep learning architecture designed to significantly improve image quality for Unmanned Surface Vehicle (USV) perception systems operating in hazy maritime conditions. By employing specialized attention modules and a unique gated fusion mechanism, MGFNet effectively enhances image detail, contrast, and color fidelity, outperforming existing methods in both quantitative metrics and visual quality.

What Problem Does This Paper Address?

Unmanned Surface Vehicles (USVs) are increasingly vital for maritime operations, but their visual perception systems struggle in hazy conditions. Haze degrades image quality by reducing resolution and contrast, obscuring crucial details, and introducing color distortions. This degradation severely impacts downstream tasks like navigation, object detection, and environmental monitoring. Existing dehazing methods, whether traditional or deep learning-based, often fall short in effectively handling the complex challenges of maritime haze, leading to incomplete restoration or undesirable artifacts. This paper aims to bridge this gap by developing a robust and effective solution for maritime image dehazing.

Methods

The core of the proposed solution is the Multi-Branch Gated Fusion Network (MGFNet), an end-to-end deep learning model. MGFNet operates by processing the hazy input image through three distinct branches, each employing a specialized residual attention module:

  • Residual Channel Attention: Focuses on mitigating color distortion caused by atmospheric light scattering.
  • Residual Spatial Attention: Enhances image contrast and improves global visibility.
  • Residual Pixel Attention: Specifically targets and recovers details in thick haze regions.

These branches, built upon a U-Net-like encoder-decoder structure, extract different aspects of the image information. Crucially, a Gated Fusion Sub-network is introduced. This sub-network adaptively learns importance weights for the feature maps from each branch. These weights are then used to linearly fuse the feature maps, allowing the network to prioritize and combine the most relevant information for generating a high-quality, haze-free image. This gated fusion approach is designed to seamlessly integrate features while preserving critical details and color information.

Datasets

The paper utilizes a combination of datasets for training and evaluation:

  • RESIDE Dataset: A portion of the public RESIDE dataset was used, comprising 1339 clear images and 13,990 synthesized hazy images. These hazy images were generated using an atmospheric scattering model with global atmospheric light values between 0.8 and 1.0, and atmospheric scattering parameters between 0.04 and 0.2.
  • Maritime Synthetic Dataset: To specifically address maritime conditions, the authors created a synthetic dataset. This involved capturing clear images of maritime surfaces and USVs using laboratory UAV photography. These clear images were then transformed into hazy images using the same atmospheric scattering model as RESIDE, with similar parameter ranges.
  • Total Dataset: The combined training set consisted of 500 RESIDE synthetic images and 500 maritime surface synthetic images. A total of 800 images were used for training, and 200 were reserved for testing.
  • Real-world Maritime Images: For qualitative evaluation and application testing (sea-skyline detection), real-world maritime images were used.

Key Insights and Ideas

  • Differentiated Feature Extraction is Key: The success of MGFNet hinges on its multi-branch architecture, where each branch is specialized to address a particular aspect of haze degradation (color, contrast, thick haze details). This is a more nuanced approach than single-branch models.
  • Adaptive Fusion Matters: The gated fusion sub-network is a critical innovation. Instead of simple concatenation or averaging, adaptively learning the importance of each branch's features allows for a more intelligent and effective combination, leading to better overall image restoration.
  • Attention for Detail: The use of residual attention modules (channel, spatial, and pixel) is crucial for focusing the network's capacity on the most informative parts of the image, especially in challenging hazy regions.
  • Maritime Specificity: By incorporating a custom maritime synthetic dataset, the authors ensure their model is well-suited for the unique challenges of this environment, which differ from general land-based dehazing.

Key Results

  • Quantitative Performance: MGFNet significantly outperformed several state-of-the-art dehazing methods (DCP, DehazeNet, AOD-Net, GCANet) on both synthetic and real-world datasets. It achieved the highest SSIM (0.9452) and PSNR (28.33) on synthetic datasets.
  • Qualitative Improvement: Visual comparisons demonstrated that MGFNet produced haze-free images with better color retention, improved contrast, and more recovered details, especially in thick haze areas, compared to other methods.
  • Sea-Skyline Detection: When applied to the task of sea-skyline detection, MGFNet achieved a detection accuracy of 95%, matching the best performing method (AOD-Net) and significantly improving upon others, particularly in hazy conditions.
  • Runtime: MGFNet's average dehazing time was 0.36 seconds, which is comparable to other deep learning methods and much faster than traditional methods, though slightly slower than AOD-Net.

Limitations and Future Work

  • Runtime Optimization: While MGFNet is efficient, the authors acknowledge that its processing speed can still be improved. Future work will focus on further lightening the network structure and optimizing its runtime for real-time applications.
  • Network Structure Optimization: The paper suggests that further optimization of the network architecture could lead to even better performance.
  • Real-time Goal: The ultimate goal is to achieve real-time dehazing for USV perception systems, which requires continued efforts in network efficiency.

Summary updated

Abstract

Image data acquired by unmanned surface vehicle (USV) perception systems in hazy situations is characterized by low resolution and low contrast, which can seriously affect subsequent high-level vision tasks. To obtain high-definition images under maritime hazy conditions, an end-to-end multi-branch gated fusion network (MGFNet) is proposed. Firstly, residual channel attention, residual pixel attention, and residual spatial attention modules are applied in different branch networks. These attention modules are used to focus on high-frequency image details, thick haze area information, and contrast enhancement, respectively. In addition, the gated fusion subnetworks are proposed to output the importance weight map corresponding to each branch, and the feature maps of three different branches are linearly fused with the importance weight map to help obtain the haze-free image. Then, the network structure is evaluated based on the comparison with pertinent state-of-the-art methods using artificial and actual datasets. The experimental results demonstrate that the proposed network is superior to other previous state-of-the-art methods in the PSNR and SSIM and has a better visual effect in qualitative image comparison. Finally, the network is further applied to the hazy sea–skyline detection task, and advanced results are still achieved.
BibTeX
@article{fan2022multi,
  author = {Yunsheng Fan and Longhui Niu and Ting Liu},
  title = {{Multi-Branch Gated Fusion Network: A Method That Provides Higher-Quality Images for the USV Perception System in Maritime Hazy Condition}},
  journal = {Journal of Marine Science and Engineering},
  year = {2022},
  doi = {10.3390/jmse10121839},
}
Figure