TL;DR
Current metrics used to evaluate underwater image enhancement are surprisingly unreliable, often rewarding over-saturated or noisy images that humans dislike. This paper benchmarks 73 different metrics and introduces TrustUIQA, a new framework revealing that even the most popular metrics can be easily "gamed" by simple edits or adversarial attacks.
What Problem Does This Paper Address?
Underwater photography is a mess of color loss, low contrast, and murky details. While many algorithms aim to "enhance" these images, evaluating them is notoriously difficult because there is no "clean" ground-truth reference for a real-world underwater shot.
To fill this gap, researchers rely on No-Reference Image Quality Assessment (NR-IQA) metrics like UIQM and UCIQE. These metrics are supposed to act as a proxy for human judgment—a higher score should mean a better-looking image. However, the field has long suspected a "metric-human gap." If we are optimizing algorithms to chase higher scores that humans actually find unappealing, we aren't making progress; we're just over-fitting to flawed math.
Methods
The authors conducted a massive evaluation of 73 metrics, categorized into hand-crafted underwater metrics, learned underwater metrics, vision-language models (like Q-Align), and general-purpose IQA tools. Their approach involved three main "stress tests":
- Within-Scene Agreement: They measured how often a metric's preference between two versions of the same scene matched human choices, using Cohen’s kappa ($\kappa$) to quantify agreement.
- Score Manipulation (Simple Edits & Attacks): They tested if basic filters (like over-saturation or sharpening) or tiny adversarial perturbations (PGD attacks) could trick metrics into giving higher scores than those given to high-quality, human-preferred enhancements.
- Generative Drift: Following the "Banana100" methodology, they used generative models (Gemini/Nano Banana) to repeatedly "replicate" or "enhance" an image. They then checked if metrics could detect the resulting noise and color shifts.
Datasets
The study utilizes a combination of existing and new data:
- Four Opinion-Score Sets: UID2021, SAUD, SAUD2.0, and SOTA, totaling over 12,000 images.
- New Paired-Comparison Study: 20 raters making 1,920 choices across 32 scenes to establish a human baseline for simple edits.
- Generative Drift Chains: 2,800 images generated through repeated editing cycles.
- TrustUIQA Benchmark: The authors' newly released benchmark (available on GitHub and Hugging Face) that aggregates these human references, edited images, and metric scores.
Key Insights and Ideas
- The "UCIQE" Problem: The widely used UCIQE metric is particularly unreliable. It is a weighted sum of global statistics (chroma spread, contrast, saturation). Because it lacks a model of what a scene should look like, it often rewards "edits" that humans find garish or noisy.
- Learned Metrics Don't Generalize: Metrics trained on specific underwater datasets perform excellently on their own "home" data but fail to generalize to images from other labs. This suggests they are learning dataset-specific biases rather than universal quality.
- Metrics are Blind to Drift: As generative models repeatedly process an image, noise increases and colors shift (generative drift). Most metrics fail to penalize this degradation; some, including UCIQE, actually show rising scores as the image falls apart.
Key Results
- Low Agreement: No metric reached the level of agreement found between two human raters ($\kappa \approx 0.41$). The best "worst-case" metric was Q-Align, with a meager $\kappa$ of 0.23.
- Gaming the System: In 91% of cases where humans rejected a specific edit, UCIQE still preferred it. Simple saturation and sharpening filters often outscored state-of-the-art enhancement methods.
- Adversarial Vulnerability: Tiny changes ($\epsilon = 2/255$), invisible to the eye, were enough to make every tested differentiable metric prefer a raw, unenhanced photo over the best human-rated enhancement.
- Drift Detection Failure: On a "replica" chain where noise grew by 2.5x, only 22 of 73 metrics consistently showed a decrease in quality scores.
Limitations and Future Work
The authors acknowledge that their human study was relatively small (20 raters from one university). Furthermore, their paired-comparison study focused on comparing raw photos to edits, rather than directly comparing different high-end enhancement algorithms against each other.
Future Directions: * Move beyond global statistics: We need metrics that incorporate semantic understanding or "naturalness" models to avoid being fooled by simple contrast/saturation boosts. * Multi-reference evaluation: Researchers should report agreement across at least two different human-rated datasets to ensure their results aren't just artifacts of one lab's bias. * Direct noise/color measurement: For generative methods, researchers should measure noise and color shift directly rather than relying solely on IQA scores.
Summary updated