UIERL: Internal-External Representation Learning Network for Underwater Image Enhancement

Zhengyong Wang, Liquan Shen, Yihan Yu, Hui Yuan

TL;DR

This paper introduces UIERL, a novel network for underwater image enhancement that tackles two key challenges: the varying quality within a single image due to depth differences and the underutilization of information from related images. By jointly learning from internal image characteristics and external related images, UIERL achieves state-of-the-art performance in restoring clarity and color to underwater scenes.

What Problem Does This Paper Address?

Underwater environments present unique challenges for image capture. Light absorption and scattering cause significant degradation, leading to color casts, low contrast, and blurred details. A critical issue is that these distortions aren't uniform across an image; regions at different depths experience varying levels of degradation. Existing methods often treat the entire image uniformly, failing to address these internal quality variations. Furthermore, underwater imaging systems (like AUVs and ROVs) often capture multiple images of similar scenes. These correlated images contain rich, complementary information that could significantly aid in enhancing a single target image, but current single-image enhancement techniques largely ignore this external context.

Methods

UIERL employs a two-stage approach:

  • Internal Representation Learning Stage: This stage focuses on addressing the region-wise quality differences within a single image. It introduces a Depth-based Region Feature Guidance (DRFG) network. DRFG first estimates the scene depth and then segments the image into regions with similar degradation patterns. A region-wise space encoder then learns features independently for each region, allowing for differentiated enhancement. This provides a more nuanced guidance for global feature learning.
  • External Representation Learning Stage: This stage leverages information from related underwater images. An External Information Extraction Network (EIEN) is proposed to mine features from these related images. Crucially, UIERL then facilitates interaction between the internal features of the target image and the external features from related images through two novel modules:
  • External-Assist-Internal (EAI) module: Uses external features to update internal features.
  • Internal-Assist-External (IAE) module: Uses internal features to update external features. This bidirectional interaction allows for a comprehensive exploitation of both internal and external information.

Datasets

The paper uses several datasets for training and evaluation:

  • UISD Dataset: A newly developed large-scale dataset containing multiple images per scene, designed to better reflect real-world underwater applications. It consists of 313 scenes (1571 images) for training and 60 scenes (271 images) for testing.
  • SQUID: A dataset of 57 underwater stereo image pairs.
  • sub-RUIE, sub-UIEB, sub-EUVP: Subsets derived from commonly used underwater image enhancement datasets, chosen for their relevance to practical applications.
  • UISD-1200: The remaining 1200 scenes from the proposed UISD dataset, used for validation.

Key Insights and Ideas

  • Depth as a Degradation Indicator: The paper highlights that scene depth is a primary driver of varying image quality in underwater scenes, offering a direct way to segment and address these internal differences.
  • Leveraging Correlation is Key: The inherent correlation between images captured in similar underwater scenes is a rich, untapped resource for improving single-image enhancement.
  • Synergistic Internal-External Learning: Simply processing internal or external information isn't enough. The proposed EAI and IAE modules demonstrate that a dynamic, bidirectional interaction between internal and external features is crucial for optimal enhancement.
  • Region-wise Enhancement Matters: Treating different regions of an image with distinct degradation patterns separately leads to more natural and effective overall enhancement compared to global processing.

Key Results

  • State-of-the-Art Performance: UIERL achieves state-of-the-art results across five benchmark datasets (SQUID, sub-RUIE, sub-UIEB, sub-EUVP, and UISD-1200) based on quantitative metrics like CCF, UIQM, UCIQE, and Edge Intensity.
  • Superior Perceptual Quality: The method consistently achieves the highest perceptual scores, indicating visually pleasing and natural-looking enhanced images.
  • Improved Color Restoration: UIERL demonstrates strong performance in color correction, as evidenced by low angular reproduction errors on the SQUID-16 dataset.
  • Qualitative Improvements: Visual comparisons show UIERL effectively corrects color casts, enhances contrast, and restores details, outperforming existing methods that often suffer from color distortion, over-saturation, or incomplete detail recovery.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of multiple related images in the external stage implies a potential increase in computational requirements compared to purely single-image methods.
  • Dataset Dependency: The effectiveness of the external stage relies on the availability of related images. In scenarios where only a single, isolated image is available, the external component would not contribute.
  • Future Work: The paper suggests that the proposed internal-external learning framework opens new avenues for underwater image enhancement research. Further exploration of more sophisticated interaction mechanisms and potentially real-time applications could be future directions.

Summary updated

Abstract

Recently, many underwater image enhancement (UIE) methods have been proposed. Although much progress has been made, they still face two issues: (1) There exists a significant region-wise quality difference in a single underwater image due to the underwater imaging process, especially in regions with different scene depths. However, existing methods neglect this internal characteristic, yielding inferior performance; (2) Due to the unique acquisition approach, underwater image acquisition tools usually capture multiple images in the same or similar scenes. The underwater images to be enhanced in practical usage are highly correlated. However, when processing a single image, existing methods ignore the rich external information provided by related images. There is still room for improvement in their performance. This paper proposes an internal-external representation learning (UIERL) network to better perform UIE tasks with internal and external information, simultaneously. In the internal representation learning stage, a depth-based region feature guidance network is designed, including a region segmentation based on scene depth to sense regions with different quality levels, and a region-wise space encoder for region-specific feature learning, which provides an effective guidance for global features to guide intra-image differentiated enhancement. In the external representation learning stage, we first propose an external information extraction network to mine rich external information from related images. Then, internal and external features interact with each other via the proposed external-assist-internal module and internal-assist-external module, fully exploiting the rich internal and external information to better enhance a single image. Extensive experiments demonstrate the superiority of our UIERL.
BibTeX
@article{wang2024uierl,
  author = {Zhengyong Wang and Liquan Shen and Yihan Yu and Hui Yuan},
  title = {{UIERL: Internal-External Representation Learning Network for Underwater Image Enhancement}},
  journal = {IEEE Trans. Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=IwjWobrH38},
}
Figure