Is Underwater Image Enhancement All Object Detectors Need?

Yudong Wang, Jichang Guo, Wanru He, Huan Gao, Huihui Yue, Zenan Zhang, Chongyi Li

TL;DR

This paper investigates whether pre-processing underwater images with enhancement algorithms actually improves object detection performance. Surprisingly, the study finds that enhancement often hinders object detection, particularly for challenging underwater scenes, and that current image quality metrics don't reliably predict detection performance.

What Problem Does This Paper Address?

Underwater object detection is vital for marine engineering and robotics, but raw underwater images are notoriously difficult to work with due to poor visibility, color distortion, and scattering. A common intuition is that enhancing these images before feeding them to object detectors should improve performance. However, there's a lack of rigorous, comprehensive studies to confirm this. This paper aims to answer two critical questions: "Does underwater image enhancement really improve underwater object detection?" and "How does it contribute (or detract) from it?" By systematically evaluating a wide range of enhancement techniques, the authors seek to clarify this relationship and guide future research.

Methods

The authors conducted an extensive empirical study. They selected 18 state-of-the-art underwater image enhancement algorithms, spanning traditional methods, CNN-based approaches, and GANs. These algorithms were used to process existing underwater object detection datasets. Then, 7 popular deep learning-based object detectors (both two-stage and one-stage, anchor-based and anchor-free) were retrained on both the original (raw) and the enhanced datasets. This resulted in a total of 133 trained models. The performance of these models was then analyzed using various quantitative metrics (like AP, TIDE) and qualitative assessments, including Class Activation Maps (CAMs) to visualize feature importance. Complementary experiments were also performed using stronger backbones and longer training schedules to see if they could mitigate any negative effects of enhancement.

Datasets

The study primarily used two datasets:

  • URPC2020: A dataset containing 5,543 underwater images with four categories: holothurian, echinus, scallop, and starfish. It was split into training and testing sets.
  • RUOD (Real-world Underwater Object Detection): A larger dataset with 14,000 images and over 74,000 bounding boxes across ten categories, including fish, echinus, corals, starfish, holothurian, scallop, diver, cuttlefish, turtle, and jellyfish.

Key Insights and Ideas

  • Enhancement is often detrimental: The most striking finding is that, contrary to intuition, most underwater image enhancement techniques suppress object detection performance compared to using raw images.
  • Hard cases are most affected: The negative impact of enhancement is most pronounced on challenging underwater scenes and "harder" samples, where it seems to introduce more noise and interference, confusing the detectors.
  • Visual quality ≠ Detection performance: Current image quality assessment metrics (like AG, EI, UIQM) don't reliably correlate with object detection performance. An image might look better but lead to worse detection.
  • Color cast isn't the main culprit: While color cast is a common underwater image problem, it's not the primary reason enhancement hurts detection. Other factors like introduced noise, edge blur, and texture corruption are more impactful.
  • Over-processing is harmful: Enhancing image properties like saturation and color richness too much can degrade detector performance.
  • Stronger detectors don't fix it: Using more powerful backbones or longer training schedules for detectors doesn't overcome the negative impact of image enhancement.

Key Results

  • Raw data wins: In almost all tested scenarios, detectors trained on raw underwater images achieved the highest Average Precision (AP) scores.
  • Performance degradation: For example, on the URPC2020 dataset, detectors retrained on enhanced images often showed a decrease in AP compared to those trained on raw images. Some enhancement methods led to AP drops of several percentage points.
  • Error analysis: TIDE analysis revealed that enhancement significantly increases background errors (detecting background as objects) and missed detections, which are the most critical factors affecting AP.
  • Qualitative examples: Visual results (Fig. 4) show that while raw images might be visually poor, detectors trained on them often correctly identify objects, whereas enhanced images lead to missed detections or false positives.

Limitations and Future Work

  • Limitations: The study acknowledges that the URPC2020 dataset has a relatively simple background, and the RUOD dataset, while larger, might still have limitations. The analysis focuses on pre-processing; end-to-end joint optimization of enhancement and detection is a separate area.
  • Future Work: The authors suggest several directions:
  • Developing enhancement algorithms with better generalization, avoiding introduced noise and color casts, and specifically targeting background interference.
  • Creating new evaluation metrics for image enhancement that better align with human perception and the requirements of downstream tasks like object detection.
  • Designing "task-oriented" enhancement methods and exploring joint optimization strategies for image enhancement and object detection, especially for low-quality image scenarios.

Summary updated

Abstract

Underwater object detection is a crucial and challenging problem in marine engineering and aquatic robotics. The difficulty is partly because of the degradation of underwater images caused by light selective absorption and scattering. Intuitively, enhancing underwater images can benefit high-level applications like underwater object detection. However, it is still unclear whether all object detectors need underwater image enhancement as preprocessing. We therefore pose the questions “Does underwater image enhancement really improve underwater object detection?” and “How does underwater image enhancement contribute to underwater object detection?”. With these two questions, we conduct extensive studies. Specifically, we use 18 state-of-the-art underwater image enhancement algorithms, covering traditional, CNN-based, and GAN-based algorithms, to preprocess underwater object detection data. Then, we retrain seven popular deep learning-based object detectors using the corresponding results enhanced by different algorithms, obtaining 126 underwater object detection models. Coupled with seven object detection models retrained using raw underwater images, we employ these 133 models to comprehensively analyze the effect of underwater image enhancement on underwater object detection. We expect this study can provide sufficient exploration to answer the aforementioned questions and draw more attention of the community to the joint problem of underwater image enhancement and underwater object detection. The pretrained models and results are publicly available and will be regularly updated.
BibTeX
@misc{wang2023is,
  author = {Yudong Wang and Jichang Guo and Wanru He and Huan Gao and Huihui Yue and Zenan Zhang and Chongyi Li},
  title = {{Is Underwater Image Enhancement All Object Detectors Need?}},
  year = {2023},
  eprint = {2311.18814},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.18814},
}
Figure