Do Underwater Image Quality Metrics Agree with Human Perception? A Benchmark with Human Choices, Simple Edits and Generative Drift

Rishii Kumar Singh, Abhinav Rajput, Dinesh Kumar VishwakarmaarXiv 2026 · arXiv

TL;DR

Current metrics used to evaluate underwater image enhancement are surprisingly unreliable, often rewarding over-saturated or noisy images that humans dislike. This paper benchmarks 73 different metrics and introduces TrustUIQA, a new framework revealing that even the most popular metrics can be easily "gamed" by simple edits or adversarial attacks.

What Problem Does This Paper Address?

Underwater photography is a mess of color loss, low contrast, and murky details. While many algorithms aim to "enhance" these images, evaluating them is notoriously difficult because there is no "clean" ground-truth reference for a real-world underwater shot.

To fill this gap, researchers rely on No-Reference Image Quality Assessment (NR-IQA) metrics like UIQM and UCIQE. These metrics are supposed to act as a proxy for human judgment—a higher score should mean a better-looking image. However, the field has long suspected a "metric-human gap." If we are optimizing algorithms to chase higher scores that humans actually find unappealing, we aren't making progress; we're just over-fitting to flawed math.

Methods

The authors conducted a massive evaluation of 73 metrics, categorized into hand-crafted underwater metrics, learned underwater metrics, vision-language models (like Q-Align), and general-purpose IQA tools. Their approach involved three main "stress tests":

  1. Within-Scene Agreement: They measured how often a metric's preference between two versions of the same scene matched human choices, using Cohen’s kappa ($\kappa$) to quantify agreement.
  2. Score Manipulation (Simple Edits & Attacks): They tested if basic filters (like over-saturation or sharpening) or tiny adversarial perturbations (PGD attacks) could trick metrics into giving higher scores than those given to high-quality, human-preferred enhancements.
  3. Generative Drift: Following the "Banana100" methodology, they used generative models (Gemini/Nano Banana) to repeatedly "replicate" or "enhance" an image. They then checked if metrics could detect the resulting noise and color shifts.

Datasets

The study utilizes a combination of existing and new data:

  • Four Opinion-Score Sets: UID2021, SAUD, SAUD2.0, and SOTA, totaling over 12,000 images.
  • New Paired-Comparison Study: 20 raters making 1,920 choices across 32 scenes to establish a human baseline for simple edits.
  • Generative Drift Chains: 2,800 images generated through repeated editing cycles.
  • TrustUIQA Benchmark: The authors' newly released benchmark (available on GitHub and Hugging Face) that aggregates these human references, edited images, and metric scores.
Table 1: Summary of the five human references used in the benchmark.

Key Insights and Ideas

  • The "UCIQE" Problem: The widely used UCIQE metric is particularly unreliable. It is a weighted sum of global statistics (chroma spread, contrast, saturation). Because it lacks a model of what a scene should look like, it often rewards "edits" that humans find garish or noisy.
  • Learned Metrics Don't Generalize: Metrics trained on specific underwater datasets perform excellently on their own "home" data but fail to generalize to images from other labs. This suggests they are learning dataset-specific biases rather than universal quality.
  • Metrics are Blind to Drift: As generative models repeatedly process an image, noise increases and colors shift (generative drift). Most metrics fail to penalize this degradation; some, including UCIQE, actually show rising scores as the image falls apart.
Figure 1: Examples where metrics (like UCIQE) rank a rejected, over-processed edit or a noisy generative frame higher than the human-preferred reference.

Key Results

  • Low Agreement: No metric reached the level of agreement found between two human raters ($\kappa \approx 0.41$). The best "worst-case" metric was Q-Align, with a meager $\kappa$ of 0.23.
  • Gaming the System: In 91% of cases where humans rejected a specific edit, UCIQE still preferred it. Simple saturation and sharpening filters often outscored state-of-the-art enhancement methods.
  • Adversarial Vulnerability: Tiny changes ($\epsilon = 2/255$), invisible to the eye, were enough to make every tested differentiable metric prefer a raw, unenhanced photo over the best human-rated enhancement.
  • Drift Detection Failure: On a "replica" chain where noise grew by 2.5x, only 22 of 73 metrics consistently showed a decrease in quality scores.
Figure 3. UCIQE prefers almost every edit, including those people reject. One row per edit of our study, sorted by human preference. Gray bar: share of the 160 votes that preferred the edit to the raw photo, with a 95% interval from resampling raters and photos together. Markers: share of the 32 photos for which the metric scores the edit above the raw photo. Yellow rows: edits that most people reject. Bars and markers are aggregate rates. Agree- ment on individual pairs is the study κ (Tab. S7 in the appendix). Per-edit values are listed in the appendix.

Limitations and Future Work

The authors acknowledge that their human study was relatively small (20 raters from one university). Furthermore, their paired-comparison study focused on comparing raw photos to edits, rather than directly comparing different high-end enhancement algorithms against each other.

Future Directions: * Move beyond global statistics: We need metrics that incorporate semantic understanding or "naturalness" models to avoid being fooled by simple contrast/saturation boosts. * Multi-reference evaluation: Researchers should report agreement across at least two different human-rated datasets to ensure their results aren't just artifacts of one lab's bias. * Direct noise/color measurement: For generative methods, researchers should measure noise and color shift directly rather than relying solely on IQA scores.

Summary updated

Abstract

Underwater image enhancement is challenging to evaluate due to the lack of clean reference images, so researchers rely on no-reference image quality assessment (NR-IQA) metrics. However, it remains unclear whether these metrics agree with human judgments. We evaluate 73 handcrafted underwater, learned underwater, vision-language and general-purpose metrics against human preferences on five references: four published opinion-score sets and a new paired-comparison study. For each metric, we measure how often it selects the same image as people when comparing versions of the same scene. We further test whether simple image edits and small adversarial perturbations raise metric scores above the enhancement people preferred, and whether scores decrease as repeated generative edits add noise. Our results show that no metric consistently matches human preferences. The best worst-case Cohen's kappa over the five references is only 0.23 (Q-Align), while two raters in our study agree with a median kappa of 0.41. The widely used UCIQE often favors edits that people reject. On a Nano Banana Pro replica chain that should leave the image unchanged, only 22 of 73 scores decrease at every checkpoint, while noise grows by 2.5 times. We also introduce TrustUIQA, a benchmark that combines the human references, our study choices, the edited, attacked and generated images, and the scores of every metric on every image. The code is available at https://github.com/TrustUIQA/TrustUIQA and TrustUIQA at https://huggingface.co/datasets/TrustUIQA/TrustUIQA.
BibTeX
@misc{singh2026do,
  author = {Rishii Kumar Singh and Abhinav Rajput and Dinesh Kumar Vishwakarma},
  title = {{Do Underwater Image Quality Metrics Agree with Human Perception? A Benchmark with Human Choices, Simple Edits and Generative Drift}},
  year = {2026},
  eprint = {2611.18429},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18429},
}
Figure