PhISH-Net: Physics Inspired System for High Resolution Underwater Image Enhancement

Aditya Chandrasekar, Manogna Sreenivas, Soma BiswasWACV 2024

TL;DR

This paper introduces PhISH-Net, a novel framework that combines a physics-based underwater image formation model (UIFM) with a deep learning approach inspired by the retinex model. PhISH-Net effectively enhances underwater images in real-time, addressing challenges like color degradation, low contrast, and blurriness, while also being computationally efficient.

What Problem Does This Paper Address?

Capturing clear and high-quality images underwater is notoriously difficult due to several factors: light absorption and scattering by water, which leads to color distortion, reduced contrast, and blurriness. These degraded images pose significant challenges for computer vision tasks like object detection and segmentation. While existing methods attempt to address this through image processing techniques or deep learning, many either ignore the underlying physics of underwater image formation or are computationally too expensive for real-time applications, especially with high-resolution images. This paper aims to bridge this gap by developing an efficient and effective method that leverages both physical principles and deep learning.

Methods

The proposed PhISH-Net framework has two main components:

  1. Backscatter Estimation (Physics-based): This step utilizes the Underwater Image Formation Model (UIFM) from the "Sea-Thru" work. The model decomposes an underwater image into a direct signal and backscatter. To estimate and remove backscatter, the method first estimates attenuation coefficients. Crucially, since depth information is often unavailable, they employ an off-the-shelf monocular depth estimation network to generate depth maps. This allows them to estimate the backscatter component and subsequently derive the direct signal.
  2. PhISH-Net (Deep Learning Enhancement): The direct signal obtained from the first step is often dimly lit, resembling low-light images. PhISH-Net tackles this by drawing inspiration from the retinex model, which decomposes an image into reflectance and illumination components. PhISH-Net predicts a three-channel illumination map, which is then used to enhance the direct signal. A key novelty here is the introduction of a Wideband Attenuation Prior. This prior ensures that the enhancement process adheres to the UIFM by constraining the predicted illumination map to follow the physical characteristics of light attenuation in water. The network is designed to be computationally efficient, performing most computations at a reduced resolution and then upsampling using a bilateral grid-based module to achieve high-resolution outputs.

The training of PhISH-Net involves a multi-component loss function that includes:

  • Reconstruction Loss: Ensures the enhanced image matches the ground truth.
  • Color Loss: Aims to match the color distribution of the enhanced image to the ground truth.
  • Smoothness Loss: Promotes locally smooth illumination maps, reflecting natural image properties.
  • Wideband Attenuation Loss: Enforces the physics-based attenuation prior.

Finally, a "Photofinishing" step is applied, involving white balancing and a multi-scale fusion technique to further refine the image quality.

Datasets

The paper uses the following datasets:

  • UIEB [40]: An underwater image dataset with 890 images.
  • EUVP [35]: A dataset containing 5550 paired dark and enhanced underwater images.
  • SQUID [10]: A dataset with 57 underwater images from diverse locations, used for evaluating cross-dataset generalization.

The authors used an 80:20 train-test split for UIEB and EUVP. For SQUID, they evaluated a model trained on EUVP.

Key Insights and Ideas

  • Bridging Physics and Deep Learning: The core insight is that combining physics-based models (like UIFM) with data-driven deep learning can lead to more robust and accurate image enhancement, especially for domains with strong physical degradation like underwater imagery.
  • Depth Estimation as a Key Enabler: The successful integration of an off-the-shelf monocular depth estimation network is crucial for overcoming the lack of depth information in many underwater datasets, enabling the physics-based backscatter removal.
  • Wideband Attenuation Prior: This novel prior is a significant contribution, ensuring that the deep learning enhancement respects the physical principles of light attenuation in water, leading to more natural and accurate results.
  • Computational Efficiency for High-Resolution Images: The use of a lightweight network and a bilateral-grid-based upsampler allows for real-time processing of high-resolution images, a critical requirement for many practical applications.

Key Results

  • Superior Performance: PhISH-Net consistently outperforms state-of-the-art (SOTA) underwater image enhancement methods on both UIEB and EUVP datasets, as evidenced by reference-based metrics like PSNR and SSIM, and non-reference-based metrics. For instance, on the UIEB dataset, PhISH-Net achieved a PSNR of 20.9197 and SSIM of 0.8559.
  • Real-time Processing: The method achieves real-time performance, with an overall runtime of approximately 0.47 seconds for a 512x512 image, significantly faster than many physics-based methods.
  • Lightweight Model: The model is remarkably lightweight, with only 0.55 million parameters.
  • Generalization Capability: Qualitative evaluation on the SQUID dataset using a model trained on EUVP demonstrates good cross-dataset generalization.
  • Ablation Study: The ablation study confirms the effectiveness of each component of the loss function, showing significant improvements in quality metrics as each loss term is added.

Limitations and Future Work

  • Depth Estimation Dependency: While they use an off-the-shelf depth estimator, the quality of the final enhancement is still dependent on the accuracy of the generated depth map.
  • Cross-Dataset Generalization (Qualitative): While qualitative results on SQUID are presented, a more extensive quantitative evaluation across diverse datasets could further validate generalization.
  • Future Work: The paper suggests that their framework offers a compelling solution for underwater image enhancement. Further exploration could involve investigating more advanced depth estimation techniques or exploring different physics-inspired priors. The authors also mention analyzing the utility of enhanced images for downstream tasks by analyzing SIFT features, which is detailed in the supplementary material, hinting at potential future research directions.

Summary updated

BibTeX
@inproceedings{chandrasekar2024phish,
  author = {Aditya Chandrasekar and Manogna Sreenivas and Soma Biswas},
  title = {{PhISH-Net: Physics Inspired System for High Resolution Underwater Image Enhancement}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure