Real-World Underwater Image Enhancement Based on Attention U-Net

Pengfei Tang, Liangliang Li, Yuan Xue, Ming Lv, Zhenhong Jia, Hongbing MaJournal of Marine Science and Engineering 2023 · DOI

TL;DR

This paper introduces AttU-GAN, a novel generative adversarial network for enhancing real-world underwater images. By integrating an attention U-Net generator with a multi-loss objective function, the model effectively filters noise and restores details, outperforming existing methods on benchmark datasets.

What Problem Does This Paper Address?

Underwater environments present significant challenges for image acquisition. Light attenuation, scattering, and absorption by water, along with the absence of natural light sources, lead to images that are often blurry, low-contrast, color-distorted (typically blue or green), and noisy. These degraded images hinder crucial underwater tasks like object detection and exploration. While various enhancement techniques exist, many struggle with the complexity and variability of real-world underwater conditions, leading to poor generalization or requiring extensive pre-processing. This paper aims to address these limitations by developing a robust and generalizable underwater image enhancement method.

Methods

The core of the proposed method is a conditional generative adversarial network (GAN) named AttU-GAN.

  • Generator: The generator is based on the Attention U-Net architecture. This choice is significant because the U-Net's encoder-decoder structure with skip connections is well-suited for image-to-image translation tasks. The key innovation here is the integration of an attention gate (AG) mechanism within the skip connections. This AG mechanism acts as a filter, allowing the network to selectively focus on relevant feature information from the encoder and suppress irrelevant noise before merging it with the decoder's features. This helps in preserving structural details and improving reconstruction quality.
  • Discriminator: The discriminator uses a Patch-GAN architecture. Unlike standard discriminators that classify an entire image as real or fake, Patch-GAN classifies overlapping patches of the image. This approach is computationally more efficient and has been shown to produce high-quality results, especially for image generation tasks.
  • Objective Function: To train the GAN effectively, the authors propose a novel, multi-loss objective function. This function combines three distinct loss components:
  • MSE Loss (Mean Squared Error): Measures pixel-wise differences, penalizing large errors.
  • MAE Loss (Mean Absolute Error): Measures absolute pixel-wise differences, less sensitive to outliers than MSE.
  • Perceptual Loss: Based on feature maps from a pre-trained VGG network, this loss encourages similarity in high-level image features between the generated and reference images, capturing structural and textural aspects. The combination of these losses, with carefully tuned weights, aims to achieve a balanced enhancement that considers global content, color, and structural information.

Datasets

  • UIEB (Underwater Image Enhancement Benchmark) dataset: This dataset, containing 950 real-world underwater images with corresponding reference images (generated by multiple enhancement methods and voted on by volunteers), was used for training, validation, and testing the proposed model. The authors split it into 710 training, 90 validation, and 90 test images.
  • EUVP (Enhanced Underwater Visual Perception) dataset: This dataset was used to validate the generalization capability of the trained model. It contains 515 paired real-world underwater images.

Key Insights and Ideas

  • Attention Gates for Underwater Enhancement: The integration of attention gates within the U-Net architecture is crucial. It allows the model to intelligently filter out noise and focus on salient features, which is particularly beneficial for the noisy and complex nature of underwater images. This is more than just a standard U-Net; it's a U-Net that learns where to look for important information.
  • Synergistic Loss Functions: Relying on a single loss function for complex image enhancement tasks can be limiting. The paper demonstrates that a carefully weighted combination of MSE, MAE, and perceptual losses provides a more comprehensive evaluation of image quality, leading to better overall results than any single loss alone. This highlights the importance of a multi-faceted approach to defining "good" image enhancement.
  • Real-World Data is Key: The authors emphasize training on real-world datasets (UIEB) rather than solely synthetic ones. This is critical for achieving good generalization, as synthetic datasets often fail to capture the full complexity of real-world underwater conditions.

Key Results

  • Superior Performance: The proposed AttU-GAN model consistently outperformed several state-of-the-art methods (Fusion, Statistics, UGAN, FUnIE, and DEA-Net) in both qualitative (visual inspection) and quantitative evaluations across multiple metrics.
  • Quantitative Metrics:
    • PSNR: Achieved the best PSNR score (21.852 dB), even surpassing the reference images in some cases.
  • SSIM: Achieved the second-best SSIM score (0.875), very close to the best.
  • UCIQE: Achieved the highest UCIQE score (0.936), indicating superior color density, saturation, and contrast.
  • UIQM: Achieved the second-best UIQM score (3.053), demonstrating strong overall underwater image quality.
  • Ablation Studies: Experiments using single loss functions (MSE Only, MAE Only) showed that the proposed multi-loss function significantly improved results, both subjectively and objectively.
  • Generalization: The model demonstrated strong generalization capabilities when tested on the unseen EUVP dataset, producing excellent enhancement results on unfamiliar data.

Limitations and Future Work

  • Computational Cost: While the Patch-GAN improves efficiency, GAN-based methods can still be computationally intensive. The authors mention that in future work, they will focus on improving the network's performance to make it "faster, lighter, and stronger."
  • Reference Image Dependency: While the model is trained end-to-end and aims for real-time enhancement, the quantitative metrics like PSNR and SSIM rely on reference images. The qualitative results and generalization on EUVP suggest robustness even without perfect references.

The paper successfully presents a robust and effective solution for real-world underwater image enhancement, paving the way for improved underwater vision applications.

Summary updated

Abstract

In recent years, with the increasingly serious problems of resource shortage and environmental pollution, the exploration and development of underwater clean energy were particularly important. At the same time, abundant underwater resources and species have attracted a large number of scientists to carry out research on underwater-related tasks. Due to the diversity and complexity of underwater environments, it is difficult to perform related vision tasks, such as underwater target detection and capture. The development of digital image technology has been relatively mature, and it has been applied in many fields and achieved remarkable results, but the research on underwater image processing technology is rarely effective. The underwater environment is much more complicated than that on land, and there is no light source underwater. Underwater imaging systems must rely on artificial light sources for illumination. When light travels through water, it is severely attenuated by water absorption, reflection, and scattering. The collected underwater images inevitably have problems such as limited visible range, blur, low contrast, uneven illumination, incoherent colors, and noise. The purpose of image enhancement is to improve or solve one or more of the above problems in a targeted manner. Therefore, underwater image enhancement technology has become one of the key contents of underwater image processing technology research. In this paper, we proposed a conditional generative adversarial network model based on attention U-Net which contains an attention gate mechanism that could filter invalid feature information and capture contour, local texture, and style information effectively. Furthermore, we formulate an objective function through three different loss functions, which can evaluate image quality from global content, color, and structural information. Finally, we performed end-to-end training on the UIEB real-world underwater image dataset. The comparison experiments show that our method outperforms all comparative methods, the ablation experiments show that the loss function proposed in this paper outperforms a single loss function, and finally, the generalizability of our method is verified by executing on two different datasets, UIEB and EUVP.
BibTeX
@article{tang2023real,
  author = {Pengfei Tang and Liangliang Li and Yuan Xue and Ming Lv and Zhenhong Jia and Hongbing Ma},
  title = {{Real-World Underwater Image Enhancement Based on Attention U-Net}},
  journal = {Journal of Marine Science and Engineering},
  year = {2023},
  doi = {10.3390/jmse11030662},
}
Figure