A learnable full-frequency transformer dual generative adversarial network for underwater image enhancement

Shijian Zheng, Rujing Wang, Shitao Zheng, Liusan Wang, Zhigui LiuFrontiers in Marine Science 2024 · DOI

TL;DR

The LFT-DGAN framework introduces a "full-frequency" approach to underwater image enhancement, using reversible convolutions to decompose images into distinct frequency bands without losing information. By combining a learnable transformer with a dual-domain discriminator (spatial and spectral), it achieves state-of-the-art color correction and detail recovery with significantly fewer parameters than competing models.

What Problem Does This Paper Address?

Capturing clear images underwater is a nightmare for computer vision. Because water absorbs and scatters light differently across the spectrum—red light disappears first, while blue and green persist—images typically suffer from severe color casts, low contrast, and hazy "fog."

While deep learning has improved things, many current models treat the image as a single block of data, often ignoring the nuances of the frequency domain. Standard downsampling in neural networks also tends to discard high-frequency details (like fine textures) that are already sparse in underwater environments. The authors aim to bridge this gap by explicitly processing different frequency components while maintaining total information integrity.

Methods

The authors propose the Learnable Full-frequency Transformer Dual Generative Adversarial Network (LFT-DGAN). Its architecture rests on three pillars:

  1. Reversible Convolutional Decomposition: Instead of standard pooling or strided convolutions that lose data, the model uses reversible blocks to split the image into low-, medium-, and high-frequency branches. Because the operation is reversible, no "natural information" is lost during the decomposition phase.
  2. Learnable Full-Frequency Transformer: This module acts as the "interactive brain" of the generator. It uses spatial and channel similarity to align features across the different frequency branches. It also employs Half Instance Normalization (HIN) to retain structural information while normalizing style, preventing the high-frequency details from being suppressed by the dominant low-frequency background.
  3. Dual-Domain Discriminator: To ensure the generated images look realistic, the authors use two discriminators. One evaluates the spatial domain (the pixels), while the other evaluates the frequency domain (the power spectrum). The frequency discriminator specifically looks at the 1D profile of the Fourier power spectrum to eliminate the "checkerboard" artifacts common in GAN-generated images.

Datasets

The model was trained and evaluated on three major benchmarks:

  • UIEB (Underwater Image Enhancement Benchmark): 890 pairs of raw and "ground truth" reference images.
  • UCCS (Underwater Color Correction Set): Subsets specifically categorized by color bias (Blue, Green, and Cyan) to test color restoration.
  • UIQS (Underwater Image Quality Set): Five distinct levels of visibility used to test contrast enhancement.

Key Insights and Ideas

  • Frequency Decoupling is Key: By separating the optimization objectives—focusing on color in the low-frequency branch and texture in the high-frequency branch—the model avoids "frequency conflicts" where the network struggles to learn two very different types of data simultaneously.
  • Information Preservation: The use of reversible networks is a clever workaround for the "information bottleneck" found in traditional CNNs. It ensures that even the fainted underwater textures are available for the enhancement process.
  • Spectral Supervision: The "aha moment" here is that a GAN can be "fooled" in the spatial domain but rarely in the frequency domain. Forcing the generator to match the power spectrum of a clear image results in much more natural-looking textures.

Key Results

  • Quantitative Superiority: On the UIEB dataset, LFT-DGAN achieved a PSNR of 24.45 and an SSIM of 0.8284, outperforming SOTA models like USLN and URSCT.
  • Efficiency: Despite its complexity, the model is remarkably lean. It uses only 3.3 million parameters, compared to 69.3 million for the STSC model, and boasts an average inference time of 15.23ms, making it viable for real-time applications.
  • Generalization: Beyond standard underwater enhancement, the model showed strong performance in "de-raining," "de-fogging," and removing motion blur, suggesting the frequency-decomposition approach is robust for various types of image degradation.

Limitations and Future Work

The authors acknowledge that while the model performs excellently on optical camera data, it hasn't been tested on sonar imaging, which is a primary sensor for deep-sea exploration and presents entirely different noise characteristics.

Future work is directed toward mutual transfer learning, aiming to create a model that can bridge the gap between different underwater sensors (optical vs. acoustic). Additionally, they suggest investigating how this enhancement affects high-level downstream tasks like underwater object detection and robotic path planning.

Summary updated

Abstract

Underwater applications present unique challenges such as color deviation, noise, and low contrast, which can degrade image quality. Addressing these issues, we propose a novel approach called the learnable full-frequency transformer dual generative adversarial network (LFT-DGAN). Our method comprises several key innovations. Firstly, we introduce a reversible convolution-based image decomposition technique. This method effectively separates underwater image information into low-, medium-, and high-frequency domains, enabling more thorough feature extraction. Secondly, we employ image channels and spatial similarity to construct a learnable full-frequency domain transformer. This transformer facilitates interaction between different branches of information, enhancing the overall image processing capabilities. Finally, we develop a robust dual-domain discriminator capable of learning spatial and frequency domain characteristics of underwater images. Extensive experimentation demonstrates the superiority of the LFT-DGAN method over state-of-the-art techniques across multiple underwater datasets. Our approach achieves significantly improved quality and evaluation metrics, showcasing its effectiveness in addressing the challenges posed by underwater imaging. The code can be found at https://github.com/zhengshijian1993/LFT-DGAN .
BibTeX
@inproceedings{zheng2024learnable,
  author = {Shijian Zheng and Rujing Wang and Shitao Zheng and Liusan Wang and Zhigui Liu},
  title = {{A learnable full-frequency transformer dual generative adversarial network for underwater image enhancement}},
  booktitle = {Frontiers in Marine Science},
  year = {2024},
  doi = {10.3389/fmars.2024.1321549},
}
Figure