Orientation-Aware Masking for Image Quality Metrics

People do not see all edge directions equally. Horizontal and vertical detail is resolved better than diagonal detail, a phenomenon called the oblique effect. The visual-search literature adds a second, less obvious fact. A field of horizontal or vertical elements suppresses itself strongly and goes quiet, while a field of diagonal elements stays "noisy." So an odd element stands out against a cardinal background and gets buried in an oblique one. Image quality metrics, the functions that score how much a distorted image differs from its reference, ignore both facts. PSNR, SSIM, the filtered-error metric PAMSE, and deep-feature metrics such as LPIPS and DISTS all weight an error the same way whether it falls on a window frame or on diagonal foliage.

We propose to test, and then exploit, one specific hypothesis. Part of the gap between metric scores and human opinion comes from orientation-dependent masking. The visibility of an artifact depends on how cardinal or oblique the reference content around it is, and on whether the artifact brings oblique energy into cardinal structure or the reverse. The contribution has three parts. First, a residual analysis that checks whether today's metric errors line up with local orientation statistics. Second, a rotation counterfactual human study that measures the effect directly. Third, a small divisive-normalization front end with only a few parameters, whose inhibition is allowed to break rotational symmetry. We compare it against an identical symmetric version, so any gain can be attributed to the orientation asymmetry and nothing else.

Core Claim

Our claim is that a quality metric should down-weight an error when the reference neighborhood contains oblique texture, and should treat a cardinal-context error as more visible. This weighting should depend on the reference rather than on the pair. That makes the metric deliberately asymmetric: $D(\text{ref},\text{dist}) \neq D(\text{dist},\text{ref})$.

The neural motivation comes from three models of visual search, which play different roles in this proposal. VS-SAIM weights cardinal Gabor channels above oblique ones. It explains the search asymmetry through strong mutual inhibition among cardinal distractors and weak inhibition among oblique ones. We reuse that mechanism, but apply it to natural images and to error visibility rather than to bar arrays and reaction times. Li's V1 model derives pop-out from lateral excitation and inhibition. It openly assumes rotational symmetry and therefore cannot produce the oblique asymmetry; our ablation is built around exactly that missing term. Del Rosario et al. show that wide-reaching lateral inhibition in mouse V1 acts divisively on contrast sensitivity, which motivates a divisive rather than subtractive form.

Gupta et al. supply both motivation and a constraint. ImageNet-trained networks reproduced human search asymmetries in 5 of 6 experiments, but diverged from humans on line orientation, and the biases reversed when the network was trained on rotated images. That is why we doubt the VGG and AlexNet backbones inside LPIPS-style metrics carry the human cardinal bias. It is also why "train the backbone on rotated data" works as a clean control.

The Metric Component

Let $E_{\theta,s}(x)$ be the response of an oriented filter, at orientation $\theta$ and scale $s$, to the difference image at location $x$. Let $R_{\phi,s}(x)$ be the oriented energy of the reference at orientation $\phi$. The visibility map is

$$ V(x)=\sum_{\theta,s}\frac{w_\theta\,|E_{\theta,s}(x)|^2}{\sigma^2+\sum_{\phi}k_{\theta\phi}\,(g*R_{\phi,s})(x)}. $$

In this expression, $w_\theta$ is orientation sensitivity, $k_{\theta\phi}$ is how strongly reference energy at $\phi$ masks errors at $\theta$, $g$ is a wide spatial pooling kernel, and $\sigma$ is a constant.

The symmetric version constrains $w$ to be constant and $k$ to depend only on $|\theta-\phi|$. The asymmetric version frees the cardinal entries, which adds about four parameters. $V$ can be pooled into a standalone score. It can also be used as a spatial weight on LPIPS or DISTS feature differences, so the orientation term is tested on top of strong deep metrics and not only against MSE. Parameters are fit on one dataset, KADID-10k, and frozen for all others.

Research Program

Stage 1: residual analysis. Using PIQ implementations, we compute per-image rank residuals (human rank minus metric rank) for PSNR, SSIM, LPIPS and DISTS on PIPAL, TID2013 and KADID-10k. We regress these residuals on two quantities: a reference "cardinality index" (the share of oriented energy within ±15° of horizontal or vertical), and the oblique fraction of the error energy. Distortion type and overall error magnitude are entered as covariates. If no orientation term survives these controls, the proposal's premise is weak for current benchmarks. We will report that outcome as a result.

Stage 2: rotation counterfactual. Rotation is the cleanest way to change orientation content while keeping everything else fixed. We take a few hundred reference/distorted pairs from PIPAL and TID2013, chosen to span the cardinality index. We show them upright and rotated by 45° inside a circular aperture, which removes frame cues. Observers make pairwise judgments, aggregated into scores as in PIPAL.

The oblique effect predicts that rotating cardinal-rich content makes its artifacts less visible, and that oblique-rich content shifts the other way. The model makes a quantitative per-image prediction of this change. The rotation-symmetric ablation predicts no change by construction. Deep metrics change somewhat under rotation, but for reasons unrelated to human vision. The headline test is correlation with the human score change, not with absolute scores.

Stage 3: benchmarks. We report SROCC and PLCC on LIVE, CSIQ, TID2008, TID2013 and KADID-10k. We also report on PIPAL's GAN-distortion subset, where PIPAL showed that traditional metrics anti-correlate with human judgment. GAN textures are a natural case for oblique masking.

Stage 4: metric hacking. Following Siniukov et al., who inflated LPIPS by up to 36.8% and DISTS by up to 34.5% with preprocessors that humans rejected, we train the same DiffJPEG-based preprocessors on Vimeo90K. We first measure where in orientation space the inflating perturbations put their energy. We then retrain the preprocessors against LPIPS weighted by $V$ and check whether the achievable inflation drops.

Baselines

The naive reference point is plain MSE/PSNR together with SSIM, run through PIQ on identical splits.

The strongest classical competitor is PAMSE, which applies an isotropic Gaussian low-pass filter as a stand-in for the contrast sensitivity function. It raised TID2008 SROCC from 0.7718 (MSE) to 0.9162. It is the closest parameter-light competitor because it models sensitivity but not orientation. We also build an anisotropic PAMSE, a Fourier-domain filter with an attenuated oblique band, fit on KADID-10k with the same parameter budget as our model. Our standalone score must beat both on TID2008 and TID2013 to count as progress.

On GAN distortions, the strongest reported number in the file is LPIPS-Alex with BlurPool at SROCC 0.5135 on PIPAL GAN distortions (up from 0.4862), reported by PIPAL. LPIPS weighted by $V$, built on the same BlurPool backbone, must exceed this with no backbone retraining. DISTS will be run through PIQ; no reported PIPAL number for DISTS was found on Lacuna.

The strongest recent full-reference metric is BiRQA, which combines SSIM maps, local binary patterns and color differences in a bidirectional pyramid. It reports SROCC 0.988 on LIVE and 0.979 on CSIQ, and its authors note weaker results on TID2013 geometric distortions. We do not expect to beat it on LIVE or CSIQ, where it has little headroom. The meaningful comparisons are TID2013 per-distortion-type gains and the Stage 2 rotation test. No per-dataset PIPAL score for BiRQA was found on Lacuna.

The key ablation is the rotation-symmetric version of our own module: same filters, same fitting data, same number of free parameters re-allocated. Any gain of the asymmetric model over it is the evidence that cardinal asymmetry matters. A second control re-trains the LPIPS backbone on 45°-rotated ImageNet, following Gupta et al. If PIPAL correlation is unchanged, whatever orientation bias the backbone carries is irrelevant to current scores.

Why not just…?

Why not just fine-tune LPIPS or DISTS on PIPAL and let the data absorb any orientation effect? This is the most serious threat. Upright training data does contain orientation effects implicitly, and a fine-tuned metric may match ours on in-distribution correlation. The rotation counterfactual is where the two can separate. A fine-tuned metric has never seen rotated content, so it has no particular reason to predict how human scores change under 45° rotation. An explicit orientation-masking term predicts that change before seeing the human data. If the fine-tuned metric predicts the rotation change equally well, we will conclude that explicit modeling is unnecessary.

Why not just add an anisotropic contrast-sensitivity prefilter to PAMSE? This would capture the sensitivity half of the oblique effect, and it is included as a matched-budget baseline. What it cannot represent is context. A reference-dependent masking term means the same diagonal error is invisible in grass and obvious across a facade. If the anisotropic prefilter matches the full model in Stages 1–3, the masking claim is wrong and the simpler filter should be adopted.

Why not just use BlurPool or a multimodal scorer like Q-Align? BlurPool addresses shift sensitivity and is orthogonal to our module, so we stack the two rather than compete. Q-Align converts mean opinion scores into text levels for a large multimodal model and generalizes well out of distribution. Its authors note that downscaled inputs limit fine-grained artifact resolution, and it is not a cheap differentiable loss for training restoration networks. We will still run it on the rotation study as a reference point, because world knowledge might predict human rotation effects for reasons we do not model.

Decisive Checks

The idea stands or falls on two results. The first is whether Stage 1 finds an orientation term in metric residuals that survives controls for distortion type and magnitude. The second is whether the asymmetric model predicts the per-image human score change under rotation significantly better than its symmetric twin and the fine-tuned LPIPS. Benchmark gains on TID2013 and PIPAL GAN distortions over 0.5135, and reduced hacking inflation confirmed by human ratings, are secondary checks that only matter if the first two hold.