Simple image-quality scores face a dilemma. Raw pixel error (mean squared error, MSE) counts sensor-like noise as heavily as a lost edge, so it disagrees with human viewers. The well-known fix is to blur both images with a Gaussian before comparing them. That fix discards noise, but it also discards the sharpness of edges, so a blurred reconstruction and a crisp one start to look alike. We propose a different pre-filter. Before measuring error, smooth both images along the edges of the clean reference image and never across them. Noise gets averaged away. A softened, broken, or missing edge still produces a full-strength difference, because the step profile across the edge is left intact.
The idea borrows a mechanism from edge detection, where the same tension appears. Taking derivatives amplifies noise, while isotropic smoothing (blurring equally in all directions) destroys localization. Elongated filters aligned with the boundary resolve this, and their noise response falls as the filter gets longer. We make that filter part of a full-reference metric and training loss. Because the filter's orientation is computed from the clean reference only, the resulting error stays a quadratic, convex function of the distorted image. It therefore keeps the cheapness and differentiability that make MSE attractive. The test is whether one fixed setting can score noise and blur correctly at the same time, which isotropic smoothing cannot do.
Core Claim
Define the reference image $R$ and the distorted image $D$. Let $A_R$ be a linear smoothing operator whose kernel at each pixel is an elongated mean oriented along the local edge direction of $R$, and isotropic in flat regions of $R$. We define
$$ E(R, D) = \frac{1}{N}\,\lVert A_R\,(R - D)\rVert_2^2 , $$
where $N$ is the number of pixels. Since $A_R$ depends only on $R$, $E$ is convex and quadratic in $D$, and its gradient is just $\frac{2}{N} A_R^\top A_R (D-R)$.
The claim is specific. Isotropic Gaussian pre-filtering forces a trade-off through its single width $\sigma$: a large $\sigma$ helps on noise distortions and hurts on blur distortions. Reference-steered oriented smoothing removes most of that trade-off. Along an edge of length $L$, zero-mean noise variance shrinks roughly as $1/L$, which is the length-dependent noise response that Galun et al.'s Difference of Oriented Means (DOM) analysis proves for elongated filters. The transverse step that blur and edge displacement alter is never averaged.
Where this sits in prior work
The direct starting point is Perceptual Fidelity Aware MSE (PAMSE). PAMSE shows that adding a high-pass "structural" term to MSE improves human correlation only when the term is given negative weight, which reduces to MSE on Gaussian-smoothed images. That reaches a Spearman rank correlation of 0.9162 on TID2008, against 0.7718 for raw MSE. PAMSE motivates our proposal, since it shows that discounting high frequencies matches human judgment. It also constrains the design: adding gradient losses naively makes the metric worse. We keep PAMSE's form and replace only the isotropic operator.
What Hinders Perceptual Quality of PSNR-oriented Methods? pulls the other way. It argues that pixel losses spend their effort on the roughly 90% of image area that is flat, and it boosts high frequencies through its contrastive loss, DECLoss. That boost assumes clean inputs, and with noisy inputs it would amplify grain along with edges. Our operator gives a principled way to resolve the tension between the two papers: it emphasizes structural error relative to noise without any explicit high-pass boost.
The oriented-filter mechanism comes from DOM, which uses elongated means at multiple lengths and orientations, computed hierarchically. A related idea is Tobias Alt's integrodifferential anisotropic diffusion, in which coarse-scale structure steers diffusion along boundaries. Both were built to detect edges in noisy images, and neither has been used as a perceptual error operator.
On the deep-metric side, the Shift-Tolerant Perceptual Similarity Metric fixes aliasing in LPIPS, a widely used learned similarity score computed from pretrained-network features. It cuts the rate at which a 1-pixel shift flips the metric's ranking on BAPPS from 2.81% to 0.57%. That work addresses translation, not the separation of noise from structure, which is our target. Scoot achieves strong human agreement on face sketches using block co-occurrence statistics, but it is neither differentiable nor general.
Research Program
We first build $A_R$. Orientation and an "edge confidence" are estimated from $R$ with a structure tensor (a local summary of gradient directions) at an integration scale tied to structure size, following the feature-centered structure tensor scale-space. At each pixel we choose an oriented box mean of length $L$ and narrow width. $L$ grows with edge confidence and is cut at junctions, where the tensor's two eigenvalues are comparable. Pixels with low confidence receive an isotropic mean. The DOM hierarchical construction makes this cost roughly $O(N\log L_{\max})$, where $L_{\max}$ is the longest filter. All filter parameters are fixed once on a held-out split.
We then evaluate in three settings. The first treats $E$ as a quality metric against human opinion scores. The second uses controlled noise-versus-blur probes. The third uses $E$ as a training loss for super-resolution from noisy low-resolution inputs, where the DECLoss tension is sharpest.
A known risk is structured artifacts such as JPEG blocking. Steering from the clean reference helps here, because block boundaries do not appear in $R$ and so do not create spurious orientations. However, low-frequency blocking in flat regions may survive the isotropic branch. We report per-distortion results rather than hiding this.
Baselines
The naive reference point is raw MSE/PSNR on the same images. Its reported TID2008 Spearman correlation is 0.7718 (PAMSE).
The strongest analytic competitor is PAMSE itself, with Gaussian pre-filtering and a reported 0.9162 Spearman correlation on TID2008. It is the most direct competitor because it shares our functional form and our convexity, and differs only in the operator. To keep the comparison fair, PAMSE's $\sigma$ gets exactly the same tuning budget as our filter parameters: one global value selected on a held-out split. We also report PAMSE's best $\sigma$ chosen separately for each distortion type, as an oracle upper bound for isotropic filtering. Beating PAMSE means two things. Our metric must exceed its global-$\sigma$ correlation on TID2008, and it must come within the oracle's per-type correlations on both the noise and blur subsets using a single setting. For LIVE and CSIQ, no reported PAMSE correlation value was found on Lacuna, so we recompute all metrics ourselves.
The strongest learned competitor is the shift-tolerant LPIPS. It reports 76.97% two-alternative forced-choice (2AFC) accuracy on CLIC, against 73.68% for LPIPS, plus the BAPPS flip rate above. We run its released weights unchanged on BAPPS, CLIC and TID2008. Matching its accuracy on BAPPS while staying convex and parameter-light would be a meaningful result. Exceeding it on the noise subsets is the claim we actually expect to win.
For the training-loss setting, the baselines are L1, PAMSE, the gradient-guided loss in the SPSR code, and DECLoss. DECLoss reports LPIPS 0.093 on Urban100 at 4× with RRDB on clean inputs. With noisy inputs, no reported result was found on Lacuna for any of these losses. All are trained with the same network, data, iterations and loss-weight sweep.
The key ablation isolates the ingredient that makes our operator distinct. We swap $A_R$ for an operator steered by the distorted image instead of the reference, and separately for generic edge-preserving smoothing (anisotropic diffusion applied to both images). This tells us whether reference steering, rather than anisotropy alone, carries the gain.
Why not just…?
Why not just tune PAMSE's Gaussian width, or average several widths? This is the real threat. Our first experiment answers it directly by sweeping $\sigma$ and plotting the noise-subset and blur-subset correlations against each other. If a single $\sigma$, or a multi-scale Gaussian average, lies on or near our point, the anisotropy buys nothing and we would report that. Our prediction rests on the arithmetic of the two cases. An isotropic filter cannot reduce noise variance by a factor $L$ without also spreading the edge step over about $\sqrt{L}$ pixels. The oriented filter does not pay that cost.
Why not just use edge-preserving smoothing such as anisotropic diffusion on both images? Generic edge-preserving smoothing steers by each image's own gradients. A noisy distorted image therefore receives its own corrupted orientations, so its noise is partly preserved as "structure", and the error stops being convex in $D$. We test this through the ablation that steers the operator by the distorted image or uses diffusion applied to both. If generic diffusion matches us, the contribution narrows to "anisotropic pre-filtering helps", and we would state it that way.
Why not just compare edge maps from a strong detector like BDCN? BDCN reaches 0.828 ODS F-measure on BSDS500, but its output is a thick probability map that needs non-maximum suppression, a step that discards everything except the local peak of each edge ridge. It is also not trained for noise invariance, and it is a non-convex network loss. Comparing binarized edge maps with distance transforms is also flawed: Liu and Haralick show that distance transforms report 0 false alarms where one-to-one assignment finds 42. We include "MSE between BDCN probability maps" as a metric baseline, so the comparison is settled empirically.
Why not just use a robust deep metric? The shift-tolerant LPIPS addresses aliasing, but deep features are known to be texture-biased (Wu). Nothing in the file shows that deep features separate grain from fine structure. The noise-subset comparison above decides this, and we accept that the deep metric may win on generic BAPPS distortions.
Proposed Experiments
The first experiment is the decisive one. On TID2008, LIVE and CSIQ we compute Spearman and Pearson correlation with human opinion scores, overall and per distortion type. We also produce the PAMSE $\sigma$-sweep frontier between the noise and blur subsets and mark our single-setting point on it. The predicted outcome is a point clearly beyond the Gaussian frontier.
The second is a controlled probe on BSDS500 images. We create matched-MSE pairs, one with additive Gaussian or Poisson noise and one with Gaussian blur or a 1–2 pixel edge displacement. We measure how often each metric ranks the blurred version as worse. We also run BAPPS 2AFC accuracy and the 1-pixel flip rate, and we report honestly that $A_R$ does not by itself confer transverse shift tolerance.
The third trains EDSR/RRDB super-resolution models on DIV2K with clean high-resolution targets and low-resolution inputs carrying noise at three levels. Each loss is used both alone and as an auxiliary term. We evaluate on Urban100 and BSD100 with PSNR and LPIPS, plus BSDS-style edge F-measure of a fixed detector on the outputs. The edge measure uses one-to-one matching, so thick or hallucinated edges are penalized, in line with Liu and Haralick. We complement this with a small pairwise human preference study. Success means our loss yields fewer amplified-grain artifacts than DECLoss and sharper edges than PAMSE at equal PSNR.
References
- Multiscale Edge Detection and Fiber Enhancement Using Differences of Oriented Means
- Perceptual Fidelity Aware Mean Squared Error
- What Hinders Perceptual Quality of PSNR-oriented Methods?
- Shift-Tolerant Perceptual Similarity Metric
- Scoot: A Perceptual Metric for Facial Sketches
- Feature-Centered First Order Structure Tensor Scale-Space in 2D and 3D
- Bi-Directional Cascade Network for Perceptual Edge Detection.
- Assignment Problem in Edge Detection Performance Evaluation
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference