Teaching Quality Models to Tell "Unusual" from "Damaged"

Blind image quality models score a photo without seeing a clean reference. They work by assuming that undistorted photographs follow stable statistical regularities, called natural scene statistics: heavy-tailed gradient histograms, scale-invariant correlations, and camera sensor noise. Blur, noise, and compression break these regularities in predictable ways. The same models fail badly on screenshots, computer graphics, microscopy, mammograms, and satellite images. Those images also break natural statistics, but because they come from different imaging processes, not because they are damaged, so the model reports a crisp screenshot as badly degraded. Energy-based out-of-distribution detection scores how far an input lies from the training data using the log-sum-exp of a network's output logits, the "free energy". It is a cheap way to make a model flag inputs it should not trust. It has its own known blind spot: it is largely insensitive to shifts in low-level appearance rather than in content.

We propose to train a quality model whose free energy separates two things that natural-scene statistics alone cannot. The energy should stay low for natural photographs, including damaged ones. It should rise for images whose acquisition process is non-photographic. The training uses no real non-natural images. Instead, the outlier set for energy-bounded training is built from synthetic transformations that break the physics of photographic acquisition: removing sensor noise, rendering perfectly sharp vector edges, piecewise-flat shading, low-count speckle noise, and scale changes. The standard distortion families are explicitly kept on the in-distribution side. The bet is that this pushes acquisition statistics into the directions the model's output head actually uses, so the energy can see a covariate shift it would otherwise ignore. The quality model can then abstain on screen content or electron micrographs instead of confidently scoring them as broken.

Core Claim

The NSS literature documents a hypersensitivity: models calibrated on photographs read benign non-natural structure as distortion. Image Quality Assessment in the Modern Age synthesizes this failure for screen content, where text edges are mistaken for ringing. Subjective Quality Assessment for Images Generated by Computer Graphics documents it for rendered images. Statistical Evaluation of No-Reference Image Quality Assessment Metrics for Remote Sensing Images documents it for satellite imagery. Perceptual Evaluation of Scanning Electron Microscopy Imaging documents it for electron microscopy. These papers motivate the work: they show the gap exists but offer no way for a model to know its input is outside its domain.

The energy literature documents the opposite insensitivity. Semantic or Covariate? A Study on the Intractable Case of Out-of-Distribution Detection proves that, for linear heads, weights on feature directions orthogonal to the class-separating subspace shrink toward zero. Logit-based detectors, including the energy score, therefore cannot see shifts confined to those directions. Their dog-breed experiment collapses to near 50% AUROC.

This result constrains our design. Post-hoc energy on an ordinary quality model will not work, because acquisition statistics are exactly the kind of covariate the head discards. The same proof also suggests the remedy. The theorem's premise is that the training loss gives no reason to use those directions. An energy-bounded loss whose outliers differ from inliers only in acquisition statistics supplies such a reason.

The new element is the outlier set. It is built from analytical NSS knowledge rather than collected images, and it is designed to sit on the far side of a specific line: photographic and damaged versus non-photographic and intact. Prior energy work draws no such line. Prior NSS work draws the line nowhere at all.

Research Program

The quality model is a standard blind-IQA backbone. Its quality output is discretized into $K$ ordinal bins, so it produces logits $f(x)\in\mathbb{R}^K$ and a free energy $$E(x) = -T\log\sum_{k=1}^{K} e^{f_k(x)/T},$$ with temperature $T$. The predicted quality score is the expected bin value. Training combines the usual quality loss on natural-image MOS data with the energy-bounded hinge from Energy-based Out-of-distribution Detection, which supplies the method: $$L = L_{\text{quality}} + \lambda\Big(\mathbb{E}_{x\sim D_{\text{nat}}}[\max(0,E(x)-m_{\text{in}})]^2 + \mathbb{E}_{x\sim \mathcal{A}(D_{\text{nat}})}[\max(0,m_{\text{out}}-E(x))]^2\Big).$$ Here $D_{\text{nat}}$ is the natural-image training set, pristine and distorted. $\mathcal{A}$ is the family of acquisition-breaking transforms applied to natural images. $m_{\text{in}}$ and $m_{\text{out}}$ are energy margins, and $\lambda$ weights the energy term.

The substantive work is designing $\mathcal{A}$. Each transform targets one regularity that photographs share and non-photographic domains violate:

  • Noise removal with gradient flattening. Denoising plus color-region flattening removes sensor noise and continuous texture, mimicking renders and screen content.
  • Synthetic overlays. Compositing text and vector shapes with aliasing-free edges mimics interfaces.
  • Substituted noise. Replacing Gaussian-like sensor noise with low-count Poisson or multiplicative speckle mimics electron microscopy, ultrasound, and adaptive-optics frames.
  • Contrast and channel changes. Low-contrast grayscale remapping mimics scientific detectors.
  • Scale changes. Strong rescaling alters the scale-invariant correlations.

For every transform we will measure the subband statistics the NSS literature uses, for example the generalized-Gaussian shape parameters discussed in Quality Modeling Under A Relaxed Natural Scene Statistics Model. The goal is to verify that the transform moves those statistics in a direction that the in-distribution distortions (blur, JPEG, white noise, and so on) do not.

Where a transform overlaps an in-distribution distortion, it is dropped rather than argued for. Color quantization overlaps TID2013's quantization distortion, for instance. This overlap check is the honest core of the design. If acquisition breaks and damage cannot be separated in statistic space, the method has nothing to learn.

The analysis component tests the mechanism behind the claim. We fit the semantic/covariate decomposition of Long et al. to the penultimate features: semantic directions spanned by differences between quality-bin means, covariate directions as the complement. We then measure how much head weight falls on the directions that predict NSS statistics, with and without the energy term. The claim predicts that this weight is near zero for a plain quality model and substantially nonzero after training.

Baselines

The naive reference point is the plain quality model with no abstention. Every image gets a score, and we report what happens to correlation with human ratings when screen content, CGI, and scientific images are mixed into the test pool. This quantifies the cost of the problem.

The first detector is post-hoc energy on the same quality-bin logits, with no retraining. On CIFAR-10 with a WideResNet, the post-hoc energy score lowers FPR at 95% TPR from 48.49% (softmax confidence) to 35.59%, as reported in Energy-based Out-of-distribution Detection. We expect it to fail here for the reason Long et al. give. We also run MaxLogit from Scaling Out-of-Distribution Detection for Real-World Settings, which raised ImageNet-1K AUROC from 84.6% to 87.2% over softmax confidence.

The strongest competitor is energy-bounded learning with a real auxiliary outlier pool, as in the original paper. That paper reports an average FPR95 of 3.32% on CIFAR-10, versus 8.53% for Outlier Exposure. We run it with the identical backbone, loss, margins, and tuning budget, swapping only $\mathcal{A}(D_{\text{nat}})$ for a pool of real non-target images, such as texture and scene datasets disjoint from all test domains. If real outliers do as well on unseen non-natural domains, the synthetic-physics construction adds nothing.

Two non-logit references cover the other families. The first is a density model on classical NSS features of the BRISQUE/MSCN type. The second is a Gaussian-mixture density on deep features in the style of GMM-GIQA from GIQA: Generated Image Quality Assessment.

The key ablation isolates the loss from the data. It trains a separate binary "photographic versus transformed" head on exactly $\mathcal{A}(D_{\text{nat}})$, with matched compute. A second ablation runs leave-one-transform-family-out training.

The headline task is distinguishing pristine non-natural images from distorted natural ones, and there is no reported result found on Lacuna for that task on SIQAD or QACS versus LIVE or TID2013. Beating the strongest baseline therefore means a margin we fix in advance: at least 5 AUROC points over energy-bounded learning with real outliers on the hard-pair split, averaged over held-out domains, with no more than a 0.01 drop in in-domain SRCC.

Why not just…?

Why not just train a binary domain classifier on screenshots versus photos? That needs labeled non-natural data from each target domain, which the NSS literature says is the bottleneck. It would also not transfer from screen content to electron microscopy. The more serious version uses our synthetic transforms with a binary head, and that is a real threat: the gain may come entirely from $\mathcal{A}$ rather than from the energy formulation. The binary-head ablation settles this. If it matches, we will report that the contribution is the outlier construction alone, and that energy only provides a single score shared with the quality head.

Why not just threshold classical NSS feature likelihoods? Natural-scene statistics deviate under both damage and domain change. A likelihood threshold will flag heavily blurred photographs together with crisp screenshots. The NSS-density baseline's hard-pair AUROC measures this directly, and we expect it near chance on that split even if it is strong on easy splits.

Why not just use unsupervised domain adaptation, as for screen content? No-Reference Screen Content Image Quality Assessment With Unsupervised Domain Adaptation aligns natural and screen-content features with MMD and ranking losses, reaching PLCC 0.865 on SIQAD. It requires unlabeled target images and produces scores rather than abstentions, so it cannot tell a user that a mammogram is outside its competence. We include it as a quality-correlation reference on SIQAD, not as a detector.

Why not just a generative score model? Multiscale Score Matching for Out-of-Distribution Detection reaches 99.1% AUROC on CIFAR-10 versus SVHN, but it needs a separately trained score network and detects any novelty, including damage. We run it on the hard-pair split to test whether it too confuses damage with domain change.

Proposed Experiments

Training data. Training uses LIVE and TID2013 natural images with MOS.

Held-out non-natural test domains. No test domain is seen in any form:

Metrics. Detection uses AUROC and FPR95 on two splits. The easy split is all natural versus all non-natural. The hard split is the most distorted natural tertile versus the pristine non-natural tertile; it is the decisive one. The downstream payoff uses a risk-coverage curve: SRCC and PLCC of quality predictions on accepted images as the abstention threshold varies, over a mixed pool.

Secondary check. To connect to the standard energy benchmark, we train the same acquisition-breaking outliers into a CIFAR-10 classifier. We then compare its FPR95 on the usual outlier sets against the reported 3.32%. The aim is not to win there, only to confirm that the construction does not harm semantic out-of-distribution detection.

Decisive outcomes. The proposal is supported if three things hold:

  • Hard-split AUROC clears the pre-registered margin over real-outlier energy training on held-out domains.
  • Leave-one-family-out shows each held-out real domain is caught mainly when its matching physical violation is in $\mathcal{A}$.
  • The head-weight probe shows NSS-predictive directions gaining weight only under the energy term.

The proposal is refuted if the binary-head ablation matches it, or if the hard-split gains vanish once overlapping distortion families are removed from $\mathcal{A}$.