A Two-Stage Framework for Distortion Information Estimation and Underwater Image Restoration

Jianming Liu, Congzheng Wang, Chuncheng Feng, Lei Liu, Wanqi Gong, Zhibo Chen, Libin Liao, Chang FengPhotonics 2025 · DOI

TL;DR

This paper introduces DR-Net, a novel two-stage deep learning framework for restoring distorted underwater images. By first estimating and correcting spatial distortions with DE-Net and then deblurring and refining details with IR-GAN, DR-Net achieves state-of-the-art performance on single underwater frames, crucial for marine resource acquisition and exploration.

What Problem Does This Paper Address?

Underwater imaging is notoriously challenging due to the complex interplay of water's optical properties, light absorption and scattering, and dynamic environmental factors like turbulence and water fluctuations. These phenomena lead to significant image degradation, including blurriness, low contrast, and intricate distortions that traditional image processing techniques struggle to fully address. Existing deep learning methods often tackle these issues in an end-to-end fashion, which can lead to unstable distortion correction and incomplete detail recovery. This paper aims to improve underwater image restoration by decoupling the problem into distinct stages, specifically addressing spatial distortions first, then handling blur and detail regeneration.

Methods

The proposed solution, DR-Net, is a two-stage framework:

  • Stage 1: Distortion Estimation Network (DE-Net): This stage focuses on correcting spatial distortions. It employs a hybrid architecture that combines a U-Net for local feature extraction with a Transformer encoder for global feature modeling. This allows for a comprehensive understanding of distortion patterns. A Dual-Attention Block (DA-Block) is integrated into the U-Net's skip connections to refine feature representations. DE-Net estimates a distortion map (specifically, a displacement vector field) and uses a Spatial Transformer Network (STN) to generate a distortion-corrected intermediate image.
  • Stage 2: Image Restoration Generative Adversarial Network (IR-GAN): This stage addresses image blurring and regenerates lost details. It utilizes a Generative Adversarial Network (GAN) architecture, specifically a variant of pix2pix, with a generator (G) and a discriminator (D). The IR-GAN takes the distortion-corrected image from DE-Net, along with the estimated distortion information, as input. The GAN's adversarial training helps to deblur the image and recover fine details that might have been lost during the distortion correction process.

The overall training objective for DR-Net is a weighted combination of three loss functions:

  • Content Loss (L1 loss): Enforces pixel-wise similarity between the restored and ground-truth images.
  • Adversarial Loss (LSGAN): Stabilizes training and encourages realistic image generation.
  • Perceptual Loss: Uses a pre-trained VGG16 network to measure feature-level similarity, improving the generalization of restoration results.

Datasets

The primary dataset used for training and evaluation is the LiSet [14], which is derived from ImageNet and specifically curated for underwater image restoration. It comprises 324,452 images from 1000 ImageNet classes, with 5 images per class for a validation set of 5000 images.

Additional datasets used for evaluation include:

  • TianSet [12]: Contains underwater image sequences.
  • JamesSet [19]: A diverse dataset with various image categories.
  • ThapaSet [20]: A collection of synthetic images.
  • NianSet [21]: Another synthetic dataset.

Key Insights and Ideas

  • Decoupling Distortion and Deblurring: The core insight is that treating spatial distortion and blur as separate problems, tackled sequentially, leads to superior results compared to end-to-end approaches. This allows each stage to focus on its specific task more effectively.
  • Transformer for Distortion Estimation: Leveraging the global feature modeling capabilities of Transformers in DE-Net is crucial for accurately capturing complex and varied distortion patterns in underwater images.
  • GAN for Detail Regeneration: The IR-GAN stage effectively utilizes the generative power of GANs to not only deblur images but also to reconstruct lost details, a common issue in degraded underwater imagery.
  • Single-Frame Restoration: The framework's ability to achieve high-quality restoration from a single frame is a significant advantage, making it practical for real-world applications where multi-frame sequences might not be available.

Key Results

  • DR-Net demonstrates superior performance compared to traditional methods and other state-of-the-art deep learning approaches on both synthetic and real-world underwater image datasets.
  • Quantitative evaluations on the LiSet dataset show DR-Net achieving the highest PSNR (22.5133) and SSIM (0.7828) values among the compared methods, along with the lowest LPIPS (0.2243) and DISTS (0.1321) scores.
  • Qualitative results, as shown in Figure 5 and Figure 7, indicate that DR-Net produces visually appealing images with enhanced sharpness, better distortion correction, and more natural details compared to other methods.
  • Ablation studies (Table 3) confirm the effectiveness of each component: IR-GAN alone shows decent results, but combining it with DE-Net (DE-Net+IR-GAN) and then fine-tuning (DR-Net) leads to significant improvements.

Limitations and Future Work

  • Computational Demands: The Transformer-GAN integration, while effective, incurs higher computational costs, which can be a challenge for real-time deployment on resource-constrained devices.
  • Data Dependency: Like other data-driven methods, DR-Net's performance is influenced by the diversity of the training data. Minor overfitting risks exist for rare distortion cases.
  • Extreme Conditions: In conditions beyond the training scope, restoration quality might slightly degrade.

Future work will focus on:

  • Optimizing efficiency through lightweight architectures.
  • Enhancing generalization with advanced data augmentation techniques.
  • Integrating underwater physical priors to strengthen adaptability.
  • Addressing limitations for extreme conditions and improving real-world utility.

Summary updated

Abstract

This work introduces a two-stage framework, named the Distorted underwater image Restoration Network (DR-Net), to address the complex degradation of underwater images caused by turbulence, water flow fluctuations, and optical properties of water. The first stage employs the Distortion Estimation Network (DE-Net), which leverages a fusion of Transformer and U-Net architectures to estimate distortion information from degraded images and focuses on image distortion recovery. Subsequently, the Image Restoration Generative Adversarial Network (IR-GAN) in the second stage utilizes this estimated distortion information to deblur images and regenerate lost details. Qualitative and quantitative evaluations on both synthetic and real-world image datasets demonstrate that DR-Net outperforms traditional methods and restoration strategies from different perspectives, showcasing its broader applicability and robustness. Our approach enables the restoration of underwater images from a single frame, which facilitates the acquisition of marine resources and enhances seabed exploration capabilities.
BibTeX
@inproceedings{liu2025two,
  author = {Jianming Liu and Congzheng Wang and Chuncheng Feng and Lei Liu and Wanqi Gong and Zhibo Chen and Libin Liao and Chang Feng and Chang Feng and Chang Feng},
  title = {{A Two-Stage Framework for Distortion Information Estimation and Underwater Image Restoration}},
  booktitle = {Photonics},
  year = {2025},
  doi = {10.3390/photonics12100975},
}
Figure