When people judge a batch of restored or generated images, they do not average. One warped eye in a gallery of otherwise clean faces can sink a whole system, while a hundred slightly soft textures go unnoticed. The standard ways of scoring generators ignore this. Pixel error and Fréchet Inception Distance (FID, a distance between Gaussian fits to deep-network features of real and generated images) both treat every sample's contribution linearly. The theory of the perception-distortion tradeoff, the proven fact that reconstructions cannot be both maximally faithful and maximally realistic, uses linear expectations as well. We propose to measure, from human judgments, how people turn per-image realism into a verdict on a whole set, and to build a scoring rule that matches that process.
The rule has two parts. First, a small classifier trained in a frozen feature space estimates how "real" each output looks. Formally this is a density ratio between real and generated images, the quantity that high-dimensional density estimation makes tractable only in compressed feature spaces. Second, those per-image scores are sorted and aggregated with a non-linear probability-weighting curve of the kind behavioral economics uses to describe how people overweight rare events. The weighting curve is fitted to human set-level preferences, not assumed. If humans really do overweight rare failures, this changes which point on the perception-distortion curve a practitioner should choose, and it changes how tunable restorers should be evaluated.
Core Claim
Let $s_i$ be the estimated log density ratio (log "realness") of output $i$ out of $n$. Sort the scores so that $s_{(1)}$ is the least realistic. A rank-weighted perceptual index is
$$P_w = \sum_{i=1}^{n} \left[ w\!\left(\tfrac{i}{n}\right) - w\!\left(\tfrac{i-1}{n}\right) \right] \ell\big(s_{(i)}\big),$$
where $\ell$ is a decreasing penalty and $w:[0,1]\to[0,1]$ is an increasing weighting function. The index contains the simpler options as special cases. The mean penalty is $w(p)=p$. Conditional value at risk, which averages only over the worst fraction $\alpha$ of outputs, is $w(p)=\min(p/\alpha,1)$. The inverse-S polynomial used by On Human-Aligned Risk Minimization is a third, smooth option. That paper reweighted sorted training losses in tabular regression so that a 5% minority group was protected. We reuse its weighting family for a different job: aggregating per-image realism at evaluation time.
The empirical claim is threefold. Human preferences between image sets with matched average quality but different tails are explained better by a fitted $w$ than by the mean or by conditional value at risk. The fitted $w$ has a steep region near zero, meaning rare failures are inflated. Choosing an operating point on a tunable restorer by $P_w$ yields outputs humans prefer over those chosen by FID. Because the index nests both alternatives, the shape of the fitted $w$ can be tested directly. If conditional value at risk fits as well as the smooth curve, we will report that and adopt the simpler rule.
Research Program
Per-image realism. We follow Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space With Softplus Loss. That work estimates real-versus-generated density ratios with a multilayer perceptron on frozen ResNet-34 features, trained with a bounded Softplus loss, and used them only as a post-hoc rejection filter for GAN samples. We keep the estimator but use its scores as inputs to a set-level index.
Two cautions shape this step. First, On the Quantitative Analysis of Decoder-Based Generative Models showed that kernel density estimates in high dimensions misrank models by hundreds of nats. That result rules out kernel-based per-image scores and justifies a discriminative ratio estimator. Second, a classification-pretrained feature space can be blind to local synthesis artifacts. We therefore train ratio estimators on several feature spaces (ResNet-34, Inception-V3, and a patch-level variant on 16×16 crops, the patch size EnhanceNet used for texture matching) and treat the choice as an ablation.
Human aggregation data. We build pairs of image sets from the same restorer in which the mean estimated realism is matched but the fraction of severe failures differs. This is the "same mean, different skew" construction that On Human-Aligned Risk Minimization used to motivate its weighting. Raters choose which set they would trust. We fit the parameters of $w$ (inflection point and curvature) by maximum likelihood under a logistic choice model, separately for faces and for general scenes. K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge approximates human arena rankings with a bias-corrected vision-language judge in fewer than 90 model runs. We use its judge only to pre-screen pairs, because that paper itself notes the judge misses subtle high-frequency artifacts, which are exactly the failures we study.
Re-reading the tradeoff. The Perception-Distortion Tradeoff and Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff define the perception term as a divergence between distributions. An f-divergence is itself a linear expectation of a function of the density ratio, so $P_w$ is its rank-weighted counterpart. These papers motivate the question but do not answer it: they prove a frontier exists without saying how humans weigh different parts of it.
One-Step Flow for Image Super-Resolution with Tunable Fidelity-Realism Trade-offs supplies the instrument. Its one-step distilled flow model, OFTSR, sweeps one knob $t$ from fidelity to realism. At the endpoints it reports PSNR 28.99 dB at $t=0$ and FID 14.58 at $t=1$ on DIV2K 4× super-resolution, and the authors note artifacts appear when realism is pushed too far. We trace the curve of PSNR against FID and of PSNR against $P_w$ along $t$. Our prediction is that failures in the tail grow faster than the average near $t=1$, so $P_w$ places the human-preferred point at a smaller $t$ than FID does. Face restoration is where identity-altering hallucinations are documented (Panini-Net: GAN Prior Based Degradation-Aware Feature Interpolation for Face Restoration, VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder), so we expect the effect to be largest there.
Baselines
Naive reference: FID. This is what papers report today. We compute it with the same reference sets and sample counts as every other index. Rethinking FID Through the Geometry of the Reference Dataset is the main evidence against it. That paper shows FID's response to quality is moderated by reference-set density and effective rank, which explain 70.7% and 66.1% of the variation respectively. It also shows FID can worsen as sample quality improves on MS COCO.
Linear ablation: mean realism penalty ($w(p)=p$). This is the most important ablation because it uses the identical density-ratio estimator and differs from our index only in aggregation. Any gain $P_w$ shows over it is attributable to the weighting curve alone.
Hard-tail ablation: conditional value at risk. We tune $\alpha$ over the same grid and on the same held-out human data used to fit $w$. Both rules therefore receive the same tuning budget.
Strongest existing evaluator: the corrected judge from K-Sort Eval. It is the closest existing tool to human preference. That paper reports, for FLUX.1-dev, human arena rank 5 with score 28.83 against judge rank 5 with score 28.86. We run it on our image-set pairs as a set-level judge, with the same number of images per set.
For all evaluators, beating a baseline means higher held-out accuracy at predicting human set choices, with a paired bootstrap interval excluding zero. I found no number on Lacuna for any baseline's accuracy on set-level human choices. All such figures will come from our own study.
Why not just…?
Why not just use conditional value at risk or a low percentile of realism scores? This is the real threat. On Human-Aligned Risk Minimization reports that conditional value at risk over-penalizes and degrades the majority in regression, but that result comes from training on tabular data, not from human evaluation of images. It does not settle our question. The nested parameterization does: if the fitted $w$ collapses to a truncation and conditional value at risk matches $P_w$ on held-out pairs, we will recommend the simpler rule.
Why not just report precision and recall of generated features? A precision-type score would also avoid FID's sign reversal, because it rewards per-sample quality. But precision averages: it counts the fraction of samples landing near real data. It cannot distinguish a set with a few severe failures from a set with many mild ones at equal precision. Our matched-mean, different-tail pairs are built to break exactly this case. If precision predicts those choices as well as $P_w$, the weighting is unnecessary.
Why not just ask a vision-language judge or humans per image and average? Averaging per-image ratings reintroduces the linearity we are testing. A judge rating whole sets is a baseline above, and its documented blindness to subtle artifacts is a stated limitation of K-Sort Eval. If it wins anyway, that is a useful negative result for our approach.
Proposed Experiments
The study has three settings, all with human raters on image-set pairs at matched mean realism. The first is OFTSR on DIV2K and on the real-world degradation sets RealSR and RealLQ250. The second is face restoration on FFHQ and CelebA-HQ, which are concentrated reference sets in the sense of the FID-geometry paper. The third is text-to-image outputs referenced against MS COCO, which is dispersed in the same sense. Distortion is reported as PSNR and LPIPS throughout.
There are three decisive checks.
Weighting shape. Does the fitted $w$ beat both the mean and conditional value at risk on held-out human choices? Is its curve steep near zero, meaning rare failures are inflated?
Operating point. For OFTSR, do humans prefer images at the $t$ chosen by minimizing $P_w$ at a fixed PSNR budget over images at the $t$ chosen by FID?
Feature-space sensitivity. Do the conclusions survive swapping the frozen features, including the patch-level estimator? If they hold only for one feature space, the claim reduces to a property of that network, and we will say so.
References
- On Human-Aligned Risk Minimization
- Subsampling Generative Adversarial Networks: Density Ratio Estimation in Feature Space With Softplus Loss
- On the Quantitative Analysis of Decoder-Based Generative Models
- K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge
- The Perception-Distortion Tradeoff
- Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff
- One-Step Flow for Image Super-Resolution with Tunable Fidelity-Realism Trade-offs
- Panini-Net: GAN Prior Based Degradation-Aware Feature Interpolation for Face Restoration
- VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder
- Rethinking FID Through the Geometry of the Reference Dataset