Query-Only Audits of Perceptual Quality Metrics

Learned image-quality metrics are networks that score how good an image looks. They are now used both to rank restoration and generation models and as training losses. That second use is dangerous: a model optimized against such a metric can learn to raise the score with patterns humans find no better, or worse. So far, robustness to this gaming has been measured only with gradient-based attacks, which require full access to the metric's internals. This proposal measures it instead with simultaneous perturbation stochastic approximation (SPSA). SPSA estimates a gradient from just two score queries per step, by nudging every input coordinate at once in a random ±1 pattern and comparing the two resulting scores. Because it only needs the score, the same audit applies to a published differentiable metric, a metric with non-differentiable steps inside it, or a closed scoring service.

The central claim is narrower and more testable than "SPSA for perceptual optimization." SPSA's perturbation size $c$ sets the scale at which it looks at the metric: with large $c$, the method follows the slope of a blurred version of the score rather than its exact local gradient. Defenses evaluated only against gradient attacks may flatten the local gradient while leaving the score just as easy to push from slightly farther away. We call this gradient masking. If robustness shrinks as $c$ grows under a fixed visual-change budget, that is a measurable sign of it. We will apply this test to the adversarially trained metric BiRQA and to the no-reference metrics already shown to be gameable. The output is a query-budget "gameability curve" for each metric, which tells practitioners whether that metric is safe to use as a loss.

What the two directions contribute

The perceptual-misalignment literature supplies the motivation and the targets. PIPAL built a 29k-image, 1.13M-comparison human-rated set and showed that PSNR and SSIM anti-correlate with human judgment on GAN restorations, while even LPIPS reaches only 0.4862 rank correlation on those distortions. Two papers show that learned metrics can be gamed. Universal Perturbation Attack on Differentiable NR Metrics computes fixed noise patterns, using exact gradients, that inflate seven no-reference metrics. It reports Stability Scores from −5.3 (PaQ-2-PiQ) to +2.6 (SPAQ), and its universal patterns achieve 70% of the per-image attack gain at 0.1% of the compute. BiRQA shows that undefended full-reference metrics fall to rank correlation 0.30–0.57 under white-box attack on KADID-10k. Its anchored adversarial training restores this to 0.60–0.84. Both papers are restricted to gradient attacks. The first explicitly names black-box attacks on non-differentiable metrics as open. BiRQA tested FGSM, C&W and AutoAttack, but nothing query-only.

The SPSA literature supplies the method and its known weakness. Its two-query gradient estimate has variance that grows with dimension, as the $O(n^{-1/4})$ actor rate in the mean-variance actor-critic analysis illustrates. SP-CS shows that when the gradient is sparse in a known basis, compressive-sensing recovery cuts this cost sharply: at dimension 25,000 it finished in 85.4 s, against 7,526 s for the analytic-gradient method. It also shows that the benefit disappears when the gradient is dense. The closest vision use of SPSA, Geometry Construction from Caustic Images, optimized rendered images against plain MSE and observed that lower image error did not mean better geometry. That observation is a small-scale version of the misalignment problem, but the paper never used a perceptual metric. No paper in the file runs SPSA against a perceptual metric at all.

Research Program

The attack optimizes a perturbation $\delta$ added to images so that the metric score $Q$ rises, or for full-reference metrics, so that the similarity score improves while the image moves away from the reference. The perturbation must stay inside a fixed budget: a PSNR floor relative to the clean image, matched to the budgets in the two attack papers. To control SPSA's dimension problem, $\delta$ is parameterized in a reduced basis. The basis is low-frequency DCT coefficients per color channel, at several sizes from about $10^3$ to $10^5$ coefficients. In this setting the SP-CS sparsity assumption is plausible, because published universal perturbations concentrate their energy in structured patterns, and we test that assumption directly. The SPSA update is the standard one. The quantity it actually follows is the gradient of the smoothed score

$$Q_c(\delta) = \mathbb{E}_{\Delta}\big[\,Q(\delta + c\Delta)\,\big],$$

where $\Delta$ is a random ±1 vector in the DCT basis and $c$ is the perturbation size. This holds because a symmetric difference with finite step $c$ estimates the slope of $Q_c$, not of $Q$ itself.

Running the attack at a ladder of $c$ values gives, for each metric, a curve of achievable score inflation against query count and against $c$. These curves are the audit's output. We build the attack in both an image-specific form and a universal form. In the universal form, one $\delta$ is optimized across a batch of images, matching the setting of the universal-perturbation paper so the results can be compared directly.

The metrics to audit come in three groups. The first is the differentiable no-reference metrics from the universal-perturbation paper (PaQ-2-PiQ, SPAQ, KonCept512, Nima). This group is a sanity check, because white-box ground truth exists for it. The second is full-reference metrics, LPIPS undefended and BiRQA with anchored adversarial training; this is where the masking hypothesis lives. The third is metrics that cannot be attacked by backpropagation: the same networks wrapped with a JPEG re-encode or 8-bit quantization front end, as a deployed scoring service might have, and the Human Preference Score model queried as a black box. The HPS authors flag black-box preference feedback as an open problem.

Baselines

The naive reference point is random noise of the same budget in the same DCT basis, with no optimization. It tells us how much score change is simply due to the metric's sensitivity to noise.

The strongest existing method is the white-box gradient attack with full metric access: projected gradient ascent, plus the universal-pattern attack of Shumitskaya et al. and the attacks used by BiRQA. For BiRQA on KADID-10k, the reported number is rank correlation 0.60–0.84 under white-box attack, against 0.30–0.57 for undefended metrics. For universal attacks, the reported numbers are the Stability Scores listed above. We follow BiRQA's protocol for computing rank correlation under attack and reuse the same images and PSNR budgets. Compute is matched by counting one backward pass as two forward evaluations. Both attacks get the same step-size tuning grid on a held-out split.

SPSA "beats" the white-box attack in two specific senses. On BiRQA, it must push rank correlation below 0.60 at the same budget. On the quantized-front-end metrics, white-box attack has no gradient to follow, while SPSA must still achieve inflation comparable to what it achieves on the unwrapped network.

We include two other query-only baselines. The first is natural evolution strategies (NES), which use Gaussian probes and many queries per step. The second is the simulated annealing that the caustics paper found slower than SPSA in its setting. Two ablations isolate our ingredients: SPSA directly in pixel space, which removes the reduced basis, and SPSA with the smallest $c$ only, which removes the scale ladder.

Why not just…?

Why not just transfer an attack from a differentiable surrogate metric? Transfer is cheap and needs no queries. It is a real competitor for auditing closed metrics, so we run it: perturbations crafted with white-box access to LPIPS or an undefended network are applied to each target. Transfer measures what two metrics share, not what the target itself is weak to. A metric that resists transferred perturbations but falls to SPSA would show that transfer understates the risk. If transfer matches SPSA everywhere, we will report that the cheaper tool is enough.

Why not just use white-box attacks with smoothed gradients, averaging gradients over random noise? For differentiable metrics, this optimizes the same blurred objective $Q_c$ with lower variance, and it is the most serious threat to our masking claim. We include it as a baseline wherever gradients exist. If it reproduces SPSA's results on BiRQA, then the finding is "BiRQA is masked at small scales," and SPSA's specific contribution narrows to the cases with no usable gradient. We say this up front. Those cases are the quantized front ends, closed scorers and HPS-as-a-service, and they remain uncovered by gradient methods.

Why not just use NES or another query-only optimizer? NES also follows a smoothed gradient and is a near relative. We do not claim SPSA dominates it. The comparison at equal query budgets decides which estimator to recommend. The audit's conclusions about which metrics are gameable should not depend on the choice; if they do, that is itself a finding about how reliable such audits are.

Proposed Experiments

The first check is calibration. On the four differentiable no-reference metrics, does SPSA's ranking of metrics by gameability reproduce the published Stability Score ordering, with SPAQ the most stable and PaQ-2-PiQ the least, within a query budget of $10^4$ per universal perturbation? Failure here would mean the audit is not trustworthy, and the rest of the program stops.

The second check is the decisive one: the masking test on BiRQA. We run SPSA on KADID-10k with BiRQA's protocol and budgets, across the $c$ ladder. If rank correlation under SPSA at large $c$ falls below the 0.60 floor reported for white-box attacks, while the noise-averaged gradient attack at small scale does not, that is evidence the anchored adversarial training defends local gradients rather than the score surface. If the defense holds at every $c$ and every query budget, that is stronger evidence for BiRQA than its own paper provides, and we report it as such.

The third check covers non-differentiable targets: the quantized-front-end metrics and HPS used as a black box. Here the comparison is SPSA against transfer, NES, and the random-noise floor.

The fourth check tests whether score gains are real. On a few hundred PIPAL reference images, we collect pairwise human judgments between clean and attacked images whose metric score rose by a fixed amount, following PIPAL's pairwise format. The audit is only meaningful if humans do not prefer the attacked images.

Finally, a basis study tests the SP-CS sparsity premise. We compare recovered gradients in the DCT basis against the true gradients on the differentiable metrics. If they turn out dense, the reduced basis is the wrong choice and we fall back to plain SPSA with more queries.