Local One-to-One Matching for Perceptual Image Metrics

Image quality metrics that compare a generated image to a reference location by location punish the wrong things. A super-resolution network that draws a realistic grass texture one pixel off from the reference is scored as worse than a network that outputs a smooth blur. As a result, on images produced by GANs (generative adversarial networks, generators trained against a discriminator to produce realistic detail), the standard metrics end up roughly uncorrelated or even anti-correlated with what people prefer. The usual fix is to blur the comparison, through anti-aliasing filters in the network or Gaussian smoothing before taking the error. Blurring throws away the high-frequency detail that separates a good GAN restoration from a bad one. We propose a different fix. Keep the sharp deep features, and let each feature in the distorted image search a small neighborhood of the reference for its partner. Partners are assigned by an optimal one-to-one matching, so no two distorted features may claim the same reference feature, and anything left unmatched costs a fixed penalty.

In two sentences: we turn LPIPS, a widely used metric that compares deep network features at identical coordinates, into a matching problem in which each feature may move a bounded distance and reference features cannot be shared. The bound on movement keeps a misplaced eye from being forgiven. The one-to-one rule keeps a bland or repetitive output from scoring well by matching the same easy reference patch many times.

Core Claim

The claim is that, on generative distortions, the remaining gap between deep metrics and human judgment comes largely from two things: coordinate rigidity, and the lack of a coverage requirement. Both can be fixed without low-pass filtering by a spatially bounded, capacity-constrained assignment over feature cells. This makes a sharp, testable prediction. Relative to a windowed nearest-neighbor search at the same radius, the one-to-one constraint should help most on GAN outputs that duplicate or omit texture. It should help little on classical distortions such as noise or JPEG artifacts, where features are already in register.

Three lines of prior work come close to this idea. None of them combines displacement tolerance, a spatial bound, and exclusivity.

The first line motivates the problem. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric showed that network activations compared at co-located positions track human judgments well; this is LPIPS. PIPAL then showed that this same co-located comparison breaks on GAN restorations. In PIPAL's analysis, small offsets in synthesized texture are penalized more than oversmoothing.

The second line attacks shift sensitivity architecturally. Shift-Tolerant Perceptual Similarity Metric adds anti-aliasing filters, reduces strides, and changes padding so that 1–3 pixel shifts stop flipping rankings. PIPAL likewise inserted BlurPool, a blur-then-subsample pooling layer, into LPIPS. Both keep the coordinate-locked comparison and contain no search, so they cannot absorb larger texture rearrangements. Our proposal removes the rigidity at the comparison stage instead of at the filtering stage.

The third line supplies the matching ingredient but uses it differently. The Contextual Loss treats features as an unordered bag and matches each one softly to its most similar target feature anywhere in the image. It has no exclusivity constraint and no spatial bound. That makes it permissive in exactly the two ways we want to rule out: crowding onto one target, and forgiving content in the wrong place. Hungarian assignment, the classical algorithm for optimal one-to-one matching, has been used for evaluation before, but only over short symbolic lists. Examples include extracted caption objects in ALOHa, propositions in summary pyramids (Passonneau et al.), and word alignment in SimAlign. Those works establish that global assignment beats greedy matching when partners compete. Our proposal carries that lesson to dense two-dimensional feature grids, where the open problems are scale and spatial order.

Research Program

At each network layer $l$, let $x_i$ and $y_j$ be the channel-normalized features of the distorted and reference images at grid cells with positions $p_i$ and $p_j$. Let $w_l$ be the learned per-channel LPIPS weights. The layer distance is

$$ D_l = \min_{P} \sum_{i,j} P_{ij}\left[\, \lVert w_l \odot (x_i - y_j)\rVert^2 + \lambda \lVert p_i - p_j\rVert^2 \right] + \tau\, U(P). $$

Here $P$ is a 0/1 matching matrix whose rows and columns each sum to at most one, with $P_{ij}=0$ whenever $\lVert p_i-p_j\rVert > r$. The scalar $\lambda$ is a quadratic penalty on displacement. $U(P)$ counts unmatched cells on both sides, and $\tau$ is the price of leaving a cell unmatched. The final score is the LPIPS-style sum of $D_l$ over layers, normalized by cell count.

This formula nests the existing metrics. With $r=0$ and $\tau$ large, it reduces exactly to LPIPS. Dropping the column constraint while keeping $r$ gives windowed nearest-neighbor matching. Letting $r$ grow without bound and removing the constraints approaches the Contextual Loss. Every ablation is therefore one setting change away from the method, and the radius is defined in pixels so that the tolerance means the same thing at every layer.

The spatial bound is what answers the permutation-invariance hazard that Xiaoshi Wu documents for texture-biased metrics. Displacement beyond $r$ is impossible, so a feature that moved further must go unmatched and pay $\tau$. This does not enforce full topological order, unlike the dynamic-programming monotonicity constraints discussed by Weigang et al.. We deliberately accept bounded displacement as the cheaper approximation, and we test whether it suffices.

On cost, the full Hungarian algorithm scales cubically in the number of nodes and would be hopeless on a 64×64 grid. The window, however, makes the graph sparse: roughly $(2r+1)^2$ edges per node. We will solve the exact sparse problem with a min-cost-flow or auction solver to obtain reference answers. For speed we will use an entropy-regularized, log-domain Sinkhorn approximation with a local kernel on the GPU, and we will report both the agreement between the two solvers and wall-clock time.

The first deliverable is a training-free metric built on frozen LPIPS weights. The second is a lightly trained variant in which $w_l$, $\lambda$, and $\tau$ are refit on BAPPS (a large set of human two-alternative patch similarity judgments) through the Sinkhorn relaxation. The third is an assignment layer inserted into a supervised quality model, to test whether the gain is complementary to large-scale fitting.

Baselines

The do-nothing reference is LPIPS-Alex as released, compared at identical coordinates. On the GAN-based distortions of PIPAL it reaches a Spearman rank correlation (SRCC) of 0.4862, as reported in PIPAL. We also report PSNR and SSIM from the PyTorch Image Quality library to show the floor.

The strongest training-free competitor on the shift question is the anti-aliasing family. The first member is LPIPS-Alex with BlurPool, at 0.5135 SRCC on PIPAL GAN distortions in the same paper. The second is the Shift-Tolerant Perceptual Similarity Metric, which reports a 1-pixel rank-flip rate of 0.57% versus 2.81% for LPIPS on shifted BAPPS, and 76.97% versus 73.68% accuracy on CLIC. We run both from released weights on identical image pairs at identical resolution. Beating this family means two things: a higher PIPAL GAN-subset SRCC than 0.5135, and a flip rate on shifted BAPPS no worse than 0.57%, with the shift robustness coming without any blur. On the GAN-subset correlation, a gain we consider meaningful is at least the 0.027 that BlurPool itself delivered, with bootstrap confidence intervals over reference images.

The Contextual Loss, used as a metric over the same VGG or AlexNet layers, is the existing matching competitor. We run it with its published bandwidth and also tuned on BAPPS with the same tuning budget as our two parameters $\lambda$ and $\tau$.

The ablation that isolates the key ingredient is windowed nearest-neighbor matching at the same radius. We run it in two forms: one-directional, and bidirectional (a Chamfer-style average of both directions, which adds a crude coverage term). Both share our features, weights, and radius search grid. If one-to-one assignment does not beat bidirectional nearest-neighbor matching on PIPAL GAN distortions, the exclusivity claim fails.

For the supervised comparison, the reference point is the NTIRE 2022 full-reference track winner on PIPAL. It reports SRCC 0.823 and PLCC 0.828 in NTIRE 2022 Challenge on Perceptual Image Quality Assessment. We do not expect a training-free metric to reach that number. The fair test is to take one supervised architecture we can train ourselves, train it with and without the assignment layer on identical data, schedule, and seeds, and compare both in-distribution and on held-out datasets.

Why not just…?

Why not just low-pass filter, as BlurPool or PAMSE do? Filtering does fix small shifts. But PAMSE's gains, an SRCC of 0.9162 versus 0.7718 for MSE on TID2008 in Perceptual Fidelity Aware Mean Squared Error, come on classical distortions where removing high frequencies is harmless. On GAN outputs, BlurPool moved LPIPS only from 0.4862 to 0.5135. We expect filtering and matching to diverge on texture-rich PIPAL categories, because filtering erases the very detail being judged. We will also test the combination: shift-tolerant features with our matching on top. If the combination is no better than shift-tolerant LPIPS alone, matching adds nothing beyond anti-aliasing, and we will say so.

Why not just take the minimum distance over a local window, without the one-to-one constraint? This is the real threat. Windowed nearest-neighbor matching captures most of the displacement tolerance at linear cost. Its weakness is coverage. A distorted image can match one easy reference patch from many locations, so an output that repeats one texture, or smooths over half a region, can score well. The bidirectional variant patches part of this weakness. Our decisive experiment compares all three at a matched radius and stratifies results by PIPAL distortion type. If exclusivity matters, its advantage should concentrate in GAN and texture-synthesis categories and vanish on noise and compression. If no such pattern appears, we will drop the constraint and report windowed matching as the finding.

Why not just train a large supervised model on PIPAL? Supervised fitting wins in-distribution, as the 0.823 SRCC shows. The NTIRE organizers, however, note weak generalization to unseen distortions, and BiRQA shows undefended neural metrics falling to 0.30–0.57 SROCC under white-box attacks on KADID-10k. We do not argue against supervision. We test whether the assignment layer is complementary to it, using paired training runs and cross-dataset evaluation in both directions: trained on KADID-10k and tested on PIPAL, and trained on PIPAL and tested on TID2013. If the paired model shows no out-of-distribution gain, the inductive bias is not worth its cost in the supervised setting.

Proposed Experiments

The primary test is SRCC and PLCC on the PIPAL GAN-based subset and on the full PIPAL set. It covers every baseline above, uses bootstrap confidence intervals over reference images, and sweeps radius $r$ from 0 to 16 pixels. Correlation should rise from $r=0$, which reproduces LPIPS exactly. It should then peak at a few pixels and fall at large $r$, as the metric begins forgiving genuinely misplaced content. A monotone rise with $r$ would mean the spatial bound is not doing its job.

The shift-robustness test reruns shifted BAPPS. We report rank-flip rates at 1, 2, and 3 pixels and plain BAPPS two-alternative agreement, where human agreement is 73.9% according to the LPIPS paper. We then extend the shifts beyond the 3-pixel range on which the shift-tolerant metric was designed and tested. The prediction is that our flip rate stays low at 4–8 pixel shifts, where the shift-tolerant metric's filtering is not expected to help, since it contains no search.

A controlled structural test guards against the permutation hazard. On BAPPS reference patches and PIPAL reference images, we synthetically relocate a patch by distances that span the radius, and we separately tile one texture over a region. A good metric should penalize relocation once the distance exceeds $r$, and should penalize tiling at every distance. The Contextual Loss should forgive both; windowed nearest-neighbor matching should forgive tiling. This experiment directly checks the two mechanisms in the claim.

The no-regression check reports SROCC on LIVE, CSIQ, TID2013, and KADID-10k. Where the file gives figures, we compare against BiRQA's 0.988 on LIVE and 0.979 on CSIQ. The file has no reported LPIPS numbers for these datasets, so we compute them ourselves. Classical distortions should not get worse, because features are already aligned and the matching should settle at zero displacement. We also report runtime per 512×512 pair for the exact sparse solver and the Sinkhorn solver, together with the correlation between the scores they produce.

The decisive outcome is the matched-radius comparison on the PIPAL GAN subset. The proposal succeeds if one-to-one assignment beats both bidirectional nearest-neighbor matching and shift-tolerant LPIPS, and if the gain concentrates where texture is synthesized. It fails informatively if windowed search alone accounts for the gain.