AUIE–GAN: Adaptive Underwater Image Enhancement Based on Generative Adversarial Networks

Fengxu Guan, Siqi Lu, Haitao Lai, Xue DuJournal of Marine Science and Engineering 2023 · DOI

TL;DR

This paper introduces AUIE-GAN, a novel Generative Adversarial Network (GAN) designed for adaptive underwater image enhancement. It tackles the persistent issues of color distortion and poor generalization in existing methods by incorporating a trainable self-attention mechanism and a dual-path frequency-aware discriminator, leading to more realistic and stable enhancement across diverse underwater scenes.

What Problem Does This Paper Address?

Underwater imaging is crucial for marine exploration and robotics, but the complex underwater environment (water turbidity, light scattering, absorption) severely degrades image quality. This degradation manifests as color distortion, underexposure, and blur, hindering subsequent analysis and tasks. While deep learning methods have shown promise, they often struggle with generalization to different underwater conditions and can produce unnatural-looking results. Traditional methods, relying on physical models, are often limited by their inability to adapt to the variability of the underwater environment. This paper aims to bridge this gap by developing an adaptive, robust, and visually realistic underwater image enhancement (UIE) system.

Methods

The core of AUIE-GAN lies in its novel generator and discriminator architectures, built upon a U-Net framework.

Generator: * Adaptive Self-Attention: The standard self-attention mechanism is enhanced with a trainable weight. This allows the model to dynamically balance the influence of self-attention, improving its adaptability to varying degradation levels and styles. * Self-Attention Residual Module (SARM): This module combines multi-level residuals with the adaptive self-attention mechanism for better feature recovery and extraction. * Global Feature Vectors & Self-Attention Skip Connections: These are incorporated into the U-Net's up-sampling stage to improve the model's ability to capture global image information. * Instance Normalization: Replaces Batch Normalization to mitigate weakening effects when training with small batches.

Discriminator: * Dual-Path Discriminator: This novel discriminator operates in both the time domain (standard image input) and the frequency domain (using spectrograms). This allows it to assess image authenticity from both spatial and spectral perspectives, leading to more comprehensive feedback for the generator. * Frequency Domain Loss: A new loss function is introduced to specifically address fitting in the frequency domain, aiming to improve image quality by considering spectral characteristics.

Loss Functions: A weighted fusion of multiple loss functions is employed, including hinge loss for GAN stability, L1/L2 losses, SSIM loss, color loss, perceptual loss, and the proposed frequency domain loss.

Datasets

  • Training: The model was trained on a combination of the UIEB dataset (890 pairs of real underwater images) and a portion of the SUID dataset (240 pairs of synthetically degraded images).
  • Evaluation:
    • Full-Reference: SUID dataset was used for quantitative evaluation using PSNR and SSIM metrics.
  • Non-Reference: Real underwater image datasets like Challenge60, U45, and RUIE were used for qualitative and quantitative evaluation using UIQM and UCIQE metrics.

Key Insights and Ideas

  • Adaptive Self-Attention is Key: The simple addition of a trainable weight to the self-attention mechanism significantly boosts the model's ability to adapt to diverse underwater conditions, a common failing of previous methods.
  • Frequency Domain Matters: Incorporating frequency domain analysis in the discriminator and loss functions is crucial for capturing subtle image degradations and improving overall image quality beyond what time-domain analysis alone can achieve.
  • Dual-Path Discriminator for Robustness: By considering both spatial and spectral information, the dual-path discriminator provides more robust guidance to the generator, leading to more realistic and artifact-free enhancements.
  • Multi-faceted Loss for Comprehensive Improvement: The weighted fusion of various loss functions, including the novel frequency domain loss, allows the model to optimize for multiple aspects of image quality simultaneously.

Key Results

  • Quantitative Performance: AUIE-GAN achieved state-of-the-art results on both synthetic (SUID) and real underwater image datasets.
  • On the SUID dataset, it achieved the best PSNR and SSIM, with a 1.52% improvement in PSNR and 0.02% in SSIM compared to the second-best.
  • On real datasets (Challenge60, U45, RUIE), it consistently outperformed other methods in UIQM and UCIQE metrics, showing improvements of up to 4.99% in UIQM and 2.70% in UCIQE.
  • Qualitative Performance: The enhanced images exhibited significantly improved color realism, contrast, and detail preservation, with reduced color distortion and artifacts compared to existing methods. The enhancement was also noted to be stable across different styles of degraded images.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the frequency domain paths in the discriminator, using fully-connected layers, consume significant GPU resources during training.
  • Color Bias: AUIE-GAN shows a slight bias in fitting the R channel in RGB space, leading to some results appearing slightly reddish.
  • Future Work: The authors plan to address the computational cost by exploring more efficient frequency domain processing and further investigate and mitigate the observed R-channel bias. They also intend to publish these findings in future works.

Summary updated

Abstract

Underwater optical imaging devices are often affected by the complex underwater environment and the characteristics of the water column, which leads to serious degradation and distortion of the images they capture. Deep learning-based underwater image enhancement (UIE) methods reduce the reliance on physical parameters in traditional methods and have powerful fitting capabilities, becoming a new baseline method for UIE tasks. However, the results of these methods often suffer from color distortion and lack of realism because they tend to have poor generalization and self-adaptation capabilities. Generating adversarial networks (GANs) provides a better fit and shows powerful capabilities on UIE tasks. Therefore, we designed a new network structure for the UIE task based on GANs. In this work, we changed the learning of the self-attention mechanism by introducing a trainable weight to balance the effect of the mechanism, improving the self-adaptive capability of the model. In addition, we designed a feature extractor based on residuals with multi-level residuals for better feature recovery. To further improve the performance of the generator, we proposed a dual path discriminator and a loss function with multiple weighted fusions to help model fitting in the frequency domain, improving image quality. We evaluated our method on the UIE task using challenging real underwater image datasets and a synthetic image dataset and compared it to state-of-the-art models. The method ensures increased enhancement quality, and the enhancement effect of the model for different styles of images is also relatively stable.
BibTeX
@article{guan2023auiegan,
  author = {Fengxu Guan and Siqi Lu and Haitao Lai and Xue Du},
  title = {{AUIE–GAN: Adaptive Underwater Image Enhancement Based on Generative Adversarial Networks}},
  journal = {Journal of Marine Science and Engineering},
  year = {2023},
  doi = {10.3390/jmse11071476},
}
Figure