Reference-Adaptive Sobolev Distances for Image Quality, Solved with Sparse Spectral Elements

Pixel error (mean squared error, MSE) punishes an edge shifted by one pixel twice over and rewards blur. Deep-feature metrics such as LPIPS agree much better with people, but they are expensive, non-convex, flip their judgments under one-pixel shifts, and collapse under adversarial perturbations. This proposal targets the space between the two: a quality metric that is still a quadratic form in the error image, and therefore convex, differentiable and bounded in how fast it can change, but whose weighting depends on the content of the reference image. The error is measured in a "negative Sobolev" norm, a norm that first smooths the error by solving a diffusion equation and then measures what remains. The coefficients of that diffusion equation come from the reference's local contrast and edge orientation, so errors are discounted along edges and in busy textures, where human vision is known to mask them.

Solving that equation is the hard part. It changes with every reference image, has spatially varying coefficients, and in its most useful form is fourth-order. A fast Fourier or cosine transform cannot diagonalize it, and finite-difference stencils for fourth-order terms are poorly conditioned. That is exactly the case the ultraspherical sparse spectral method was built for: it discretizes variable-coefficient differential operators as almost-banded, well-conditioned matrices. The core claim is that a handful of interpretable parameters (about ten), fitted once, give a convex metric that beats the best existing analytic convex metric on standard human-rated benchmarks. The same metric would carry a provable robustness bound that deep metrics lack, and would stay stable under sub-pixel shifts without any network changes.

Core Claim

Let $r$ be the reference image, $u$ the distorted image, and $e = u - r$ the error. Define a positive definite operator built from the reference,

$$L_r = c_r(x) - \nabla\cdot\big(K_r(x)\,\nabla\big) + \beta\,\Delta\big(K_r(x)\,\Delta\big),$$

where $c_r(x) > 0$ is a local sensitivity term that shrinks where the reference has high contrast, $K_r(x)$ is a $2\times 2$ diffusion tensor taken from the reference's local structure tensor (it smooths error more strongly along edges than across them), and $\beta \ge 0$ weights an optional fourth-order term that captures curvature-scale structure. The distance is

$$d_r(u)^2 = \langle e,\; L_r^{-1} e\rangle,$$

that is, solve $L_r w = e$ once and take the inner product of $w$ with $e$. This distance is a norm of $e$, so it is convex in $u$. Its gradient comes from the same single linear solve, $\nabla_u d_r^2 = 2w$. It is also Lipschitz by construction: $|d_r(u+\delta) - d_r(u)| \le \|\delta\|_2 / \sqrt{\min_x c_r(x)}$. That inequality gives a certified bound on how far any perturbation of a given pixel budget can move the score, which is the robustness guarantee learned metrics can only approach empirically. The mapping from reference statistics to $c_r$ and $K_r$ uses roughly ten scalar parameters, fitted once on one dataset and then frozen for evaluation on the others.

The claim is falsifiable at two levels, and they are kept separate. The scientific claim is that reference-adaptive, orientation-aware discounting of error is what closes the gap between convex analytic metrics and human judgment. The engineering claim is that the sparse spectral discretization makes this accurate, stable and fast, particularly for the fourth-order term and for sub-pixel shifts. Either claim can fail without taking the other down, and the ablations below are designed so we can tell which one failed.

Where This Sits in Prior Work

Perceptual Fidelity Aware Mean Squared Error motivates the proposal and is its direct competitor. It shows that adding linear derivative terms to MSE reduces to MSE of Gaussian-smoothed images (PAMSE), which raises Spearman correlation on TID2008 from 0.7718 to 0.9162 while remaining convex. PAMSE's smoothing is the same everywhere and in every direction, and the authors note that higher-order structural terms computed with discrete stencils are numerically fragile. Our operator can be read as a PAMSE whose blur is replaced by a content-dependent, anisotropic elliptic solve, with PAMSE recovered as the special case of constant, isotropic coefficients. That special case is one of our ablations.

The Sobolev GAN line supplies the motivation for negative Sobolev norms but a different method. Sobolev GAN constrains a neural critic's gradient norm. Sobolev Wasserstein GAN relaxes the constraint with an augmented Lagrangian and reports CIFAR-10 FID 16.75 against 18.21 for WGAN-GP. Both estimate Sobolev-type discrepancies between distributions using trained networks. Neither computes a closed-form Sobolev distance between a pair of images, and neither is tested against human quality ratings. We do both, and we train no network.

The numerical method comes from the sparse spectral literature. A fast and well-conditioned spectral method makes differentiation diagonal and basis conversion banded, handles variable coefficients through banded multiplication matrices, and reports machine-precision solutions of 10th-order equations with bounded condition numbers. The automatic solution of partial differential equations using a global spectral method extends this to two-dimensional operators. The ultraspherical spectral element method keeps the operators banded on patchwise elements, at $O(p^4)$ build cost per element of polynomial degree $p$. Images are not globally smooth, so a single global Chebyshev expansion would ring at every edge. We therefore use the element version on small pixel tiles with continuity enforced across tile boundaries. Two properties make this workable. The quantity we actually need, $w = L_r^{-1}e$, is smoother than $e$ because the operator smooths. And the coefficient fields $c_r$ and $K_r$ are computed from locally smoothed reference statistics, so a low polynomial degree per element resolves them.

Two lines of perceptual-metric work set the standard we must meet. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric establishes LPIPS and the BAPPS benchmark. Shift-Tolerant Perceptual Similarity Metric traces LPIPS's shift fragility to aliasing and repairs it with anti-aliasing blur layers inside the network. Those authors say their result is not perfectly shift-invariant. BiRQA shows that lightweight analytic features combined through a small network can be both accurate and adversarially robust, but its robustness is empirical and it weakens on geometric distortions in TID2013. We do not expect to beat these learned metrics on clean accuracy. We aim to match their robustness and shift stability with guarantees rather than training, at a small fraction of their parameter count.

One cited result cuts against us. The Perceptual Misalignment of Image Quality Metrics argues that the most damaging failures of generated images, such as broken anatomy, are global and semantic. No local masking model, ours included, will detect those. We accept this limit and restrict the claim to full-reference fidelity: distortion, restoration and compression, where a reference image exists.

Research Program

The first component is the metric itself. We will compute local contrast and the structure tensor of $r$ at a few scales. We will map them through a small parametric family, such as power-law contrast masking for $c_r$ and eigenvalue reshaping of the structure tensor for $K_r$, into coefficient fields. We will then fit those coefficients on each ultraspherical spectral element by low-degree polynomial approximation, assemble the almost-banded global system, and solve it. Fitting the roughly ten parameters is a low-dimensional search maximizing rank correlation on KADID-10k. After fitting, the parameters are frozen and the metric is evaluated on the other datasets without refitting.

The second component is analysis. We will prove positive definiteness of $L_r$ under simple constraints on the parameters, derive the Lipschitz certificate above, and characterize the metric's behavior under shifts. Because the solution lives in a continuous polynomial space, a half-pixel translation of the reference can be applied exactly to the polynomial representation, with no need to resample a pixel grid.

The third component is engineering and diagnostics. We will measure the condition number and runtime of the spectral solve as resolution and polynomial degree grow, and compare them with a finite-difference discretization of the same operator solved by conjugate gradients. We will also check how sensitive scores are to element size. A secondary, optional study uses $d_r$ as a training loss for super-resolution in the SPSR codebase (resource; art_31136c6c9b764b26bf512c265cb50ff3). There it tests whether the loss's convexity and its single-solve gradient give sharper results than MSE at comparable cost.

Baselines

The naive reference points are MSE/PSNR and SSIM, run through the PIQ library (PyTorch Image Quality) for uniform implementations. On TID2008, MSE reaches Spearman 0.7718 (from the PAMSE paper). On BAPPS 2AFC, traditional metrics such as SSIM and FSIM reach about 63% against a human ceiling of 73.9% (from the LPIPS paper). Any serious result must clear these by a wide margin.

The strongest competitor within the convex, analytic class is PAMSE. It is the right comparison because it is exactly our method with constant, isotropic coefficients. It gets the same fitting budget as ours: its smoothing width and our ten parameters are both tuned on KADID-10k only. Its reported TID2008 Spearman is 0.9162. Beating it means exceeding that figure on TID2008, and also outperforming PAMSE on CSIQ, LIVE and BAPPS in our own reruns. The file quotes no PAMSE numbers for those three datasets, so those comparisons will rest entirely on reruns under identical protocol.

Two ablations isolate our key ingredients. The first removes adaptivity: a constant-coefficient negative Sobolev norm computed exactly by a discrete cosine transform. It tests whether the benefit comes from the reference-dependent coefficients or merely from Sobolev-type smoothing. The second removes the spectral discretization: the identical operator with the identical fitted parameters, discretized by finite differences and solved by conjugate gradients. It tests whether the spectral method changes the scores or only the speed. We will run each ablation with and without the fourth-order term, since that term is where the discretizations should differ most.

The strongest learned competitors are three. LPIPS (no BAPPS 2AFC number is quoted in the file beyond the 67–69% for uncalibrated networks). Shift-tolerant LPIPS (1-pixel rank-flip rate 0.57% versus 2.81% for LPIPS, and CLIC accuracy 76.97% versus 73.68%). BiRQA (Spearman 0.988 on LIVE and 0.979 on CSIQ; under white-box attack on KADID-10k, 0.60–0.84 versus 0.30–0.57 for undefended models). We do not claim to beat them on clean accuracy. We claim to beat them on three other axes. The first is a 1-pixel rank-flip rate below 0.57% on BAPPS, extended to half-pixel shifts. The second is a white-box adversarial Spearman on KADID-10k above BiRQA's upper figure of 0.84, reported alongside our certified bound. The third is throughput on 1080p images above BiRQA's roughly 15 frames per second, measured on the same GPU. Winning on those axes while giving up only a few points of clean correlation would make the result worthwhile.

Why not just…?

Why not just blur the error with a Gaussian, as PAMSE does? PAMSE is the special case of our metric with constant, isotropic coefficients. Its own authors report that the best blur width depends on viewing conditions, and a uniform blur discounts errors across an edge as much as errors along it. If our adaptive metric fails to beat PAMSE under the same fitting budget, the central scientific claim is false. The PAMSE comparison is therefore the first decisive test, not a formality.

Why not just compute a negative Sobolev norm with the FFT or DCT? For constant coefficients this is the right tool, and it would make ultraspherical machinery pointless. That is precisely why the proposal uses reference-dependent, anisotropic coefficients. Those produce an operator that no fixed transform diagonalizes, whereas the ultraspherical method keeps it banded through its multiplication matrices (A fast and well-conditioned spectral method). The DCT ablation measures how much adaptivity is worth. If the gain is negligible, the idea should shrink to the DCT version.

Why not just discretize the same operator with finite differences and multigrid? This is the real threat. For the second-order operator, finite differences are also sparse and linear-time, and they may well produce the same scores. Our bet rests on the fourth-order term and on shifts. Finite-difference fourth-order stencils have condition numbers growing with the inverse fourth power of the grid spacing, PAMSE's authors already flag high-order stencils as fragile, and a stencil cannot represent a half-pixel shift without resampling. The discretization ablation settles the question. If finite differences match the spectral version on accuracy, conditioning and shift stability, we will report that the spectral method is unnecessary and keep the metric.

Why not just use shift-tolerant LPIPS or BiRQA? They are more accurate on clean data and will likely stay so. They cannot be certified against perturbations, cannot be minimized as a convex training loss, and still show residual rank flips under shifts. If they also match our adversarial and shift results empirically, then the advantage of our metric narrows to convexity and the certificate. We would report exactly that and nothing more.

Proposed Experiments

The decisive checks come in order. First, fit on KADID-10k and evaluate the frozen metric by Spearman and Pearson correlation on TID2008, TID2013, CSIQ and LIVE, and by 2AFC agreement on BAPPS. Success means beating PAMSE's 0.9162 on TID2008 and PAMSE's rerun scores elsewhere. Second, the DCT and finite-difference ablations, with and without the fourth-order term, report both scores and solver condition numbers. Third, shift stability: measure rank-flip rates on BAPPS under 1-pixel and 0.5-pixel shifts against the 0.57% figure from shift-tolerant LPIPS, and CLIC accuracy against 76.97%. Fourth, robustness: run white-box gradient attacks on KADID-10k at matched budgets, compare with BiRQA's 0.60–0.84, and check the certified Lipschitz bound against the observed score changes. Fifth, measure runtime on 1080p inputs against BiRQA and LPIPS on identical hardware.

The main risk is edge ringing inside spectral elements, which would show up as sensitivity to element size in the diagnostics. The fallback is smaller elements with lower polynomial degree, at the cost of some of the spectral method's accuracy advantage.