Deep perceptual metrics compare two images by measuring how far apart their internal features are in a pretrained network. They agree with people far better than pixel error does. They are also easy to fool: tiny, invisible perturbations can swing their scores wildly, and distribution metrics built on the same features react strongly to noise that people cannot see. We propose that both failures come from the same place. These metrics weight every direction in feature space about equally, but natural images vary along only a small number of directions. The long tail of low-variance directions is where invisible changes and adversarial attacks do their work. The fix we study is to shrink each feature direction by how much natural images actually vary along it, using the ridge-regression filter $\sigma_i^2/(\sigma_i^2+\lambda)$. The number of directions that survive this shrinkage, the effective degrees of freedom, then becomes a measurable "perceptual dimension" for each network layer.
The proposal rests on a theoretical result that makes this more than a reweighting trick. A random sketch, meaning a projection of features onto the span of a few random combinations of natural-image feature vectors, already applies this ridge shrinkage on average, and the number of random combinations sets the strength $\lambda$. So the sketch size is the perceptual regularizer. One integer per layer, chosen to equal the effective dimension, gives a cheap metric that is randomized, needs no human labels and has a known operator underneath. We test whether that metric matches learned perceptual metrics on human-judgment benchmarks while resisting attacks that break them.
Core Claim
Let $\phi_\ell(x)$ be layer-$\ell$ features of image $x$, and let $\Sigma_\ell$ be their covariance over a bank of natural images. Write $\sigma_i^2$ for the eigenvalues of $\Sigma_\ell$ and $\Delta = \phi_\ell(x)-\phi_\ell(y)$ for the feature difference between two images. We define the shrunk perceptual distance and its degrees of freedom as
$$D_\lambda(x,y)=\sum_\ell \Delta^\top \Sigma_\ell(\Sigma_\ell+\lambda_\ell I)^{-1}\Delta, \qquad d_{\text{eff},\ell}(\lambda_\ell)=\mathrm{tr}\big(\Sigma_\ell(\Sigma_\ell+\lambda_\ell I)^{-1}\big).$$
We make three claims, each of which can be tested and refuted.
The first is spectral. On the human-judgment benchmarks, feature differences that people notice concentrate in the top $d_{\text{eff}}$ directions of natural-image covariance. Imperceptible noise and adversarial perturbations against standard metrics put a disproportionate share of their energy in the tail.
The second is about sketching. A random projection onto the row space of $S\Phi_\ell$ reproduces $D_\lambda$ in expectation. Here $\Phi_\ell$ is the natural-image feature matrix and $S$ is a $k\times n$ Gaussian matrix. The implied $\lambda$ is the one with $d_{\text{eff},\ell}(\lambda)=k$, and averaging a handful of draws approaches the soft operator.
The third is practical. With one $\lambda$ per layer fitted on a single human-judgment benchmark, $D_\lambda$ comes close to learned-weight metrics on clean images, holds up much better under white-box attack, and transfers to other benchmarks.
Grounding in Prior Work
Two results from the sketching literature supply the method. Precise expressions for random projections derives that the expected residual projection after sketching behaves like $(\gamma A^\top A+I)^{-1}$, with $\gamma$ fixed by matching the trace to the sketch size. Equivalently, the expected projection onto the sketched subspace is the ridge shrinkage operator above, so sketch size acts as implicit regularization. That paper studies low-rank approximation and Newton methods. We reinterpret the same identity as the design principle for a perceptual metric. Its proof needs the sketch size to stay below the stable rank, which is an open regime constraint. We check empirically whether the identity holds past that point for deep features.
A Unified Theory of Random Projection for Influence Functions proves that preserving bilinear forms with $(F+\lambda I)^{-1}$ requires sketch size $m=\Omega(d_\lambda(F)/\epsilon^2)$. It also quantifies a "leakage" error for vectors outside the range of $F$. Because $\Sigma(\Sigma+\lambda I)^{-1}=I-\lambda(\Sigma+\lambda I)^{-1}$, our distance is exactly that kind of bilinear form, so the sizing bound transfers directly. Leakage gives a precise language for distortions that push features off the natural-image span.
The supporting definitions come from three papers. Random Design Analysis of Ridge Regression establishes effective dimension as the right complexity measure. An Iterative, Sketching-based Framework for Ridge Regression shows that sketches can be sized by $d_\lambda$ rather than by ambient dimension. Loss Landscapes of Regularized Linear Autoencoders shows that components below $\lambda$ collapse.
Four perceptual-metric papers motivate the work, and each leaves a specific opening. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric introduced Learned Perceptual Image Patch Similarity (LPIPS). It shows that raw deep features already beat structural similarity (SSIM) on two-alternative forced choice (2AFC, picking which of two distorted patches is closer to a reference). It then reaches near-human agreement by learning per-channel weights from human labels. Those weights are diagonal and supervised. Our operator is non-diagonal, built from unlabeled natural-image statistics, and controlled by one scalar per layer.
Perceptual Fidelity Aware Mean Squared Error (PAMSE) low-pass filters pixels as a stand-in for the human contrast sensitivity function. Our proposal is the feature-space analogue: a low-pass filter whose "frequencies" are the eigendirections of natural-image feature covariance rather than spatial frequencies.
E-LPIPS gains robustness by averaging LPIPS over random image transformations. That randomization works empirically but has no guarantee and needs many samples. Our randomization is over feature-space sketches, comes with a closed-form target operator, and needs only a few draws.
BiRQA gains robustness by switching to hand-picked analytic features and adding adversarial training. It is the robustness bar we must meet without either ingredient.
Using Skew to Assess the Quality of GAN-generated Image Features cuts Inception features from 2048 to 256 dimensions by PCA before computing third-order moments. It suggests random projections as future work but gives no basis for choosing the size. Our framework supplies that basis as a secondary application.
One line of evidence constrains the design. Distribution metrics are known to be blind to global structural errors such as extra limbs, as documented in Human Preference Score: Better Aligning Text-to-image Models with Human Preference Human Preference Score: Better Aligning Text-to-image Models with Human Preference. Rare structural defects may well live in low-variance directions, which is exactly what shrinkage suppresses. We treat this as the main way the proposal could fail and test for it directly rather than assuming it away.
Research Program
We first build the spectral object. For VGG and AlexNet layers, as used by LPIPS, we pool spatial feature vectors from a large set of clean natural images and estimate $\Sigma_\ell$. We then report each layer's eigenvalue decay and $d_{\text{eff},\ell}(\lambda)$ curve. No such characterization exists in the file.
Next comes the spectral diagnostic that tests the first claim. For the distortion types in BAPPS, TID2008 and KADID-10k, we decompose $\Delta$ into eigen-bands and measure two things: what share of energy falls beyond the top-$d_{\text{eff}}$ directions, and how that share correlates with human judgments. We repeat this for two kinds of change that people do not see. One is Gaussian noise at the imperceptible levels where the SID paper reports that FID still rises linearly. The other is white-box adversarial perturbations against uncalibrated deep-feature distance and against LPIPS.
We then fit one global $\lambda$, or one per layer, by maximizing 2AFC agreement on the BAPPS training split. From the fitted $\lambda$ we read off each layer's perceptual degrees of freedom. We implement the metric two ways: exactly, through an eigendecomposition, and by sketching, with $k_\ell=\lceil d_{\text{eff},\ell}\rceil$ random combinations averaged over $T$ draws. We measure how far the sketched version deviates from the exact one as $k$ and $T$ vary, against the $O(1/\sqrt{r})$ surrogate error predicted for stable rank $r$.
Finally, in the secondary distribution-level application, we compute the Skew Inception Distance in a sketched space of size $d_{\text{eff}}$ of Inception pool3 features. We compare that against the fixed 256-dimension PCA cutoff.
Baselines
The naive reference point is uncalibrated deep-feature distance: unit-normalized VGG or AlexNet features compared with plain $L_2$ distance, all directions weighted equally. LPIPS reports 67–69% 2AFC agreement on BAPPS for these networks without perceptual fine-tuning, against a human ceiling of 73.9% (LPIPS). Our metric uses the same backbone, so any gain is attributable to the spectral operator. On clean BAPPS 2AFC, beating 69% while using no human labels beyond one fitted scalar per layer would count as a success.
The strongest clean-accuracy competitor is LPIPS with learned linear channel weights, which the source describes as nearing the human ceiling. The file gives no exact number for it, so we will rerun it on identical splits. Matching it within about one point, while staying label-light, would count as competitive. Exceeding it is not expected.
The strongest robustness competitors are E-LPIPS and BiRQA. E-LPIPS reports 69.16% 2AFC on BAPPS and runs 10–20% slower than LPIPS (E-LPIPS). BiRQA reports SROCC (Spearman rank correlation with human scores) of 0.988 on LIVE and 0.979 on CSIQ. Under FGSM, C&W and AutoAttack on KADID-10k, it lifts SROCC from 0.30–0.57 for undefended models to 0.60–0.84 (BiRQA). We run both under the same attack suite, with the same perturbation budgets, on the same images.
Beating them means two things. On KADID-10k under attack, our metric must exceed E-LPIPS and land inside or above BiRQA's 0.60–0.84 band without adversarial training. On clean data, it must stay within a few hundredths of BiRQA's SROCC. The randomization must also not mislead the evaluation. Attacks on our sketched metric therefore average gradients over many sketch draws (expectation over transformation), so that robustness cannot come from gradients that are merely noisy.
PAMSE is the cheap analytic reference on TID2008, with SROCC 0.9162 against 0.7718 for MSE (PAMSE).
Three ablations isolate the key ingredient. Hard PCA truncation to the same $k$ tests whether soft shrinkage matters. A data-oblivious Gaussian projection of raw features, a projection that preserves distances in all directions roughly equally, tests whether filtering by natural-image statistics matters rather than dimension reduction as such. Mahalanobis whitening by $(\Sigma+\lambda I)^{-1}$, which upweights the tail, is the opposite operator. If the spectral hypothesis is right, whitening should be the least robust and least human-aligned of all the variants.
Why not just…?
Why not just truncate with PCA at a tuned dimension? This is the most serious threat. Hard truncation and soft shrinkage may perform almost identically, because both discard the same tail. If PCA matches us on accuracy and robustness, the spectral finding still holds and the contribution narrows to a principled rule for choosing the cutoff: $k=d_{\text{eff}}(\lambda^*)$ instead of an arbitrary 256. The two can still separate in two ways. A sketch needs no eigendecomposition of a large feature bank, which matters for 2048-dimensional Inception features and streaming settings. Each sketch draw is also random, which hands the attacker a moving target. The matched-$k$ PCA ablation, run under adaptive attack, settles the question.
Why not just keep LPIPS's learned channel weights and add robustness separately? Learned diagonal weights fit human labels on the training distortions, but they cannot downweight a direction that mixes channels. The adversarial results reported for E-LPIPS suggest that the vulnerable directions are not aligned with individual channels. If calibrated LPIPS combined with our shrinkage beats either alone, the two are complementary. We report that combination too.
Why not just adversarially train, as BiRQA does? Adversarial training works: 0.60–0.84 SROCC under attack. But it requires generating attacks during training and committing to hand-picked features that underperform on geometric distortions such as TID2013's radial transforms. Our method needs no adversarial training at all. If it falls clearly below BiRQA's band under attack, we will report that the spectral filter is a partial defense, not a substitute.
Why not just average over random image transformations, as E-LPIPS does? E-LPIPS randomizes in image space with no target operator and needs many samples. We will measure robustness against the number of draws $T$ for both methods at equal compute. The prediction from the trace-matching identity is that a few sketch draws suffice, because the variance around the shrinkage operator is controlled by stable rank.
Decisive Checks
We will judge the project by a few outcomes. First, the spectral claim stands or falls on whether human-judged distortion energy concentrates in the top-$d_{\text{eff}}$ band on BAPPS, TID2008 and KADID-10k while adversarial energy concentrates in the tail. If the two distributions overlap, the premise is wrong.
Second, the $\lambda$ fitted on BAPPS must transfer without refitting to TID2008, TID2013, LIVE, CSIQ, KADID-10k and PIPAL. Large drops in transfer would mean that $d_{\text{eff}}$ describes a dataset rather than perception.
Third, the sketched metric at $k=d_{\text{eff}}$ must track the exact operator, with error falling as predicted. We also map where the identity breaks once $k$ exceeds the stable rank.
Fourth, the structural risk: we report per-distortion-type SROCC on TID2013, especially geometric and local structural distortions, and we test whether shrinkage lowers sensitivity to these relative to unshrunk features. A consistent loss there would show that rare structure lives in the tail. That would argue for a layer-dependent $\lambda$ that is weak in deep semantic layers, which the per-layer fit can test.
Last, at the distribution level, SID computed in a $d_{\text{eff}}$-sized sketch should keep the property its authors report, staying flat for imperceptible noise on FFHQ and CelebA, while reducing the third-order tensor cost below the 256-dimension PCA version.
References
- Precise expressions for random projections: Low-rank approximation and randomized Newton
- A Unified Theory of Random Projection for Influence Functions
- Random Design Analysis of Ridge Regression
- An Iterative, Sketching-based Framework for Ridge Regression
- Loss Landscapes of Regularized Linear Autoencoders
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Perceptual Fidelity Aware Mean Squared Error
- E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles
- BiRQA: Bidirectional Robust Quality Assessment for Images
- Using Skew to Assess the Quality of GAN-generated Image Features
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference