Toward Sufficient Spatial-Frequency Interaction for Gradient-aware Underwater Image Enhancement

Chen Zhao, Wei-Ling Cai, Chenyu Dong, Ziqi Zeng

TL;DR

This paper introduces SFGNet, a novel underwater image enhancement (UIE) framework that leverages both spatial and frequency domain information. By fusing these domains and incorporating gradient maps, SFGNet effectively addresses complex underwater image degradations, leading to improved visual quality and detail preservation.

What Problem Does This Paper Address?

Underwater images are notoriously difficult to work with due to light absorption and scattering, resulting in poor contrast, color distortion, and loss of detail. These issues significantly hinder downstream computer vision tasks like object detection and scene understanding. While deep learning has shown promise in UIE, most existing methods primarily focus on spatial domain restoration, neglecting the rich information present in the frequency domain. This paper aims to bridge this gap by proposing a method that explicitly combines spatial and frequency domain features for more robust and comprehensive underwater image enhancement.

Methods

The proposed SFGNet operates in two stages.

Stage 1: Dense Spatial-Frequency Fusion Network (DSFFNet) This network is designed to capture sufficient spatial-frequency interaction. It comprises two key components:

  • Dense Fourier Fusion (DFF) block: This block processes information in the frequency domain. It utilizes Fast Fourier Transform (FFT) to convert the image into its frequency components (amplitude and phase), manipulates these components, and then uses Inverse FFT (IFFT) to reconstruct the image. The core idea here is that degradation information is primarily concentrated in the amplitude component, allowing for targeted enhancement.
  • Dense Spatial Fusion (DSF) block: This block operates in the spatial domain, focusing on extracting spatial features. These two blocks are interconnected through cross-connections, enabling a rich exchange of information between the spatial and frequency domains. This fusion is crucial for capturing the complex degradations present in underwater images.

Stage 2: Gradient-Aware Corrector (GAC) After the initial enhancement by DSFFNet, the GAC further refines perceptual details and geometric structures. It achieves this by:

  • Gradient Map Extraction: The Sobel operator is used to compute the gradient map of the enhanced image from Stage 1. Gradient maps are known to highlight edges and structural information.
  • Gradient Map Enhancement: A small convolutional network processes the gradient map to further enhance edge details.
  • Adaptive Fusion: The enhanced image from Stage 1 is adaptively combined with the enhanced gradient map using a learnable parameter ($\alpha$). This allows the network to dynamically adjust the contribution of gradient information to the final output, preserving fine details and structures.
  • Curricular Learning: A curricular learning strategy is employed to facilitate the convergence of the GAC, gradually increasing the complexity of the learning task.

The overall framework is trained using a combination of loss functions: a spatial-frequency interaction loss ($L_{s1}$) focusing on the amplitude component, a loss for the gradient map ($L_g$), and a perceptual loss ($L_{s2}$) that includes a VGG-based feature similarity metric.

Datasets

The paper uses two real-world underwater image datasets for training and evaluation:

  • UIEBD dataset: Contains 890 underwater images and corresponding labels.
  • LSUI dataset: A larger dataset, with 4500 images for training and 400 for testing.

Key Insights and Ideas

  • Frequency Domain Matters for UIE: The paper strongly advocates for the importance of the frequency domain in underwater image enhancement, demonstrating that degradation information is largely contained within the amplitude spectrum. This challenges the prevailing spatial-domain-only approaches.
  • Amplitude-Phase Swapping Motivation: The visual experiments showing that swapping amplitude components between a degraded image and its clean counterpart effectively swaps their visual characteristics provide a compelling intuition for why targeting the amplitude is effective.
  • Gradient Maps for Detail Preservation: The GAC's ability to leverage gradient information highlights its effectiveness in recovering fine details and sharp edges, which are often lost in traditional enhancement methods.
  • Synergy of Spatial and Frequency: The DSFFNet's design, with its cross-connections between DFF and DSF blocks, emphasizes the benefit of jointly processing spatial and frequency information for a more comprehensive enhancement.

Key Results

  • State-of-the-Art Performance: SFGNet achieves state-of-the-art performance on both the UIEBD and LSUI datasets, outperforming several leading UIE methods in terms of PSNR, SSIM, and LPIPS metrics. For instance, on the UIEBD dataset, the full model achieved a PSNR of 21.66 and an SSIM of 0.8710.
  • Visual Quality: Qualitative results presented in Figures 5 and 6 show that SFGNet generates natural-looking images with improved contrast, color balance, and preserved details compared to baseline methods.
  • Ablation Studies: Ablation studies confirm the effectiveness of individual components. The DFF and DSF blocks are crucial for spatial-frequency interaction, and the GAC significantly boosts performance by enhancing perceptual details. The curricular learning strategy and adaptive parameter $\alpha$ in GAC also contribute positively.

Limitations and Future Work

  • Computational Cost: While the paper mentions Fourier transform is efficient, the overall network architecture, especially with dense connections and multiple blocks, might still incur significant computational cost for real-time applications.
  • Generalization to Extreme Conditions: While tested on real-world datasets, the performance on extremely challenging underwater conditions (e.g., very murky water, extreme lighting) might warrant further investigation.
  • Future Work: The authors suggest that their framework provides a strong foundation for future research in UIE, particularly in exploring more sophisticated spatial-frequency interaction mechanisms and advanced gradient-based refinement techniques. The paper also implies that further optimization for real-time performance could be a valuable direction.

Summary updated

Abstract

Underwater images suffer from complex and diverse degradation, which inevitably affects the performance of underwater visual tasks. However, most existing learning-based Underwater image enhancement (UIE) methods mainly restore such degradations in the spatial domain, and rarely pay attention to the fourier frequency information. In this paper, we develop a novel UIE framework based on spatial-frequency interaction and gradient maps, namely SFGNet, which consists of two stages. Specifically, in the first stage, we propose a dense spatial-frequency fusion network (DSFFNet), mainly including our designed dense fourier fusion block and dense spatial fusion block, achieving sufficient spatial-frequency interaction by cross connections between these two blocks. In the second stage, we propose a gradient-aware corrector (GAC) to further enhance perceptual details and geometric structures of images by gradient map. Experimental results on two real-world underwater image datasets show that our approach can successfully enhance underwater images, and achieves competitive performance in visual quality improvement.
BibTeX
@inproceedings{zhao2023toward,
  author = {Chen Zhao and Wei-Ling Cai and Chenyu Dong and Ziqi Zeng},
  title = {{Toward Sufficient Spatial-Frequency Interaction for Gradient-aware Underwater Image Enhancement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp48485.2024.10448182},
}
Figure