Generative models are increasingly used to make synthetic training data for medical imaging. The scores used to choose among them, such as the Fréchet Inception Distance (FID), compare the overall feature statistics of real and generated images, and they can point the wrong way. On retinal fundus photographs, StyleGAN3 checkpoints that looked more realistic by FID produced worse glaucoma classifiers when used as training data. Swapping the generic ImageNet feature extractor for a retina-specific foundation model (RETFound) did not fix the mismatch (A Pragmatic Note on Evaluating Generative Models with Fréchet Inception Distance for Retinal Image Synthesis). The only reliable check is to train a downstream model on every checkpoint's output, and that is too expensive to run during development.
Our claim is that domain-specific pre-training does help evaluation, but only once the score stops ignoring labels. FID asks whether the cloud of generated images matches the cloud of real ones. A generator can match that cloud more closely while blurring the features that separate diseased from healthy eyes, which is the image-level version of how BLEU rewards a fluent report that drops the diagnosis. We propose a cheap, closed-form score built from the same pieces as FID: a frozen domain encoder and Gaussian fits to its features. We fit one Gaussian per diagnostic class on synthetic images, use these Gaussians as a classifier on a small labeled set of real images, and weight its errors by clinical cost. The prediction is an interaction effect. Domain features should improve agreement with downstream utility under this label-aware score but not under FID.
Core Claim and Evidence Behind It
The evidence comes from two literatures that rarely meet. In image generation, the perceptual evaluation gap shows up as pixel and distribution metrics rewarding the wrong outputs. Inpainting models that produce crisp textures lose PSNR to models that output blurry averages (Spatially adaptive multi-scale contextual attention for image inpainting). The lowest-FID model is not reliably the best one, and how badly FID fails depends on the geometry of the reference set (Rethinking FID Through the Geometry of the Reference Dataset). Wu et al. add that about 78% of distribution-metric pairs rank models with Kendall $\tau > 0.7$, so FID, KID and CMMD share the same blind spots. These papers motivate the work. Wu et al. also constrain it, because they rule out "use a better encoder" as a full fix.
In domain-specific language modeling, the field has already lived through the same failure for generated reports, and the fix that worked was not simply a better encoder. RaTEScore classifies extracted entities as abnormality or non-abnormality and penalizes mismatches in that category. On synonym-versus-antonym tests it reaches 67% accuracy against 14% for BERTScore, a metric that compares generic embeddings (RaTEScore: A Metric for Radiology Report Generation). The CRG Score drops abundant true negatives and weights misses by how rare each abnormality is. It does this because accuracy can hide majority-class collapse, as with RadFM's 0.786 accuracy alongside an F1 of 0.059 on CT-RATE (CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation). RadCliQ and RadGraph F1 likewise score clinical entities and relations rather than surface overlap (Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation, RadGraph). In both cases domain pre-trained models (PubMedBERT, BioLORD) did the extraction, and the gains came from a label-aware, cost-asymmetric scoring rule placed on top. From this line we borrow the scoring structure, prevalence-weighted asymmetric errors as in CRG, and move it to synthetic image evaluation. That setting has a real downstream task but, so far, no such score.
The Proxy
Let $\phi$ be a frozen encoder. Let $\mu_c$ and $\Sigma$ be the mean of $\phi$-features for each class $c$ and the shared covariance, both estimated on synthetic images from generator checkpoint $G$. Let $\pi_c$ be the class prior. Every real calibration image $x$ is assigned to the class $\hat c(x) = \arg\max_c \log \mathcal{N}(\phi(x);\mu_c,\Sigma) + \log \pi_c$. The checkpoint's score is
$$ U(G) = 1 - \frac{w_{FN}\,\mathrm{FNR}(G) + w_{FP}\,\mathrm{FPR}(G)}{w_{FN}+w_{FP}}, $$
where FNR and FPR are the false-negative and false-positive rates on the real calibration set. The weights $w_{FN}$ and $w_{FP}$ are set from disease prevalence following the CRG rule, so that missing a rare positive costs more than a false alarm. The cost of computing $U$ is the same as FID's: one feature pass and one covariance estimate. The only addition is a labeled real calibration set of a few thousand images, kept separate from the downstream test set. For segmentation, the same idea applies per patch: per-layer Gaussians are fit on synthetic patch features with their generated masks, and the resulting pixel classifier is scored with a cost-weighted Dice on real OCT scans.
The score answers a question FID does not ask. FID asks whether synthetic features look like real features. $U$ asks whether the class structure in the synthetic features carries over to real data. The StyleGAN3 paradox has a testable explanation under this view: later checkpoints match the marginal distribution better while the class-conditional means move closer together. We measure this directly by tracking Fisher separability between classes in the synthetic features across checkpoints, next to FID.
Research Program
We first reproduce the Wu et al. setting. This covers StyleGAN3, DDPM and Medfusion checkpoint trajectories on AIROGS (about 101,000 fundus images, referable glaucoma) and on MICCAI GOALS (OCT layer segmentation of RNFL, GCIPL and CL). For each checkpoint we compute the ground-truth utility used there, the F1 of a full classifier trained on synthetic data and the Dice of a full segmenter, both evaluated on real held-out data. This is the expensive reference everything is ranked against.
We then run a two-by-two study of encoder against statistic. The encoders are Inception-V3 trained on ImageNet and RETFound. The statistics are label-blind (FID) and label-aware ($U$, plus intermediate variants). The main analysis is the interaction: does moving from Inception to RETFound raise rank agreement with utility under $U$ while leaving FID unchanged, as Wu et al. observed? That result would answer whether domain pre-training helps generative evaluation, a question the current evidence leaves negative. If both encoders do equally well under $U$, the conclusion is that the statistic, not the representation, was the bottleneck, and that is also worth reporting.
Baselines
The naive reference point is to do nothing: take the final checkpoint, or the checkpoint with the lowest Inception FID. This is what most practitioners do. Our proxy has to choose better checkpoints, measured by the downstream F1 or Dice that the selected checkpoint yields.
The strongest existing evaluation method on this exact question is the family of distribution metrics run on the domain backbone: FID, KID and CMMD on RETFound features, as in Wu et al. They are the closest competitor because they already use domain pre-training and the same cost budget. No correlation coefficient is reported for them in the file (no reported result found on Lacuna for a numeric FID–F1 correlation on AIROGS). The paper reports only a statistically significant negative correlation for StyleGAN3 and non-significant correlations for diffusion models on GOALS. We compute all of them on the identical sample sets, sample counts and feature layers used for $U$. Beating them means a positive Kendall $\tau$ with downstream utility across checkpoints, with a bootstrap confidence interval that excludes their $\tau$. The bar is a $\tau$ that is significantly positive on both generator families, and in particular on StyleGAN3, where FID's correlation has the wrong sign.
Three ablations isolate the key ingredient. Class-conditional FID computes the Fréchet distance separately for each class on RETFound features and then averages. It is label-aware but does not transfer to real data and does not weight costs, so it tests whether conditioning alone is enough. Unweighted $U$ sets $w_{FN} = w_{FP}$ to isolate the CRG-style cost asymmetry. A linear-probe variant replaces the Gaussian classifier with logistic regression trained on synthetic features, testing whether the closed form gives up any accuracy.
The most dangerous competitor is truncated downstream training: train the real classifier or segmenter for a fraction of an epoch on a subsample of synthetic data. We run it at several budgets and plot rank agreement against GPU-seconds. $U$ wins only if it reaches comparable $\tau$ at substantially lower cost, roughly FID's cost.
Why not just…?
Why not just compute FID on a domain foundation model's features? Wu et al. ran exactly this experiment with RETFound and found that the mismatch with diagnostic utility remained. Their redundancy result, about 78% of metric pairs with $\tau > 0.7$, suggests that switching among label-blind distances changes little. Our two-by-two design repeats this cell as a control.
Why not just run short downstream training? This is a real threat, because it measures utility directly, only with more noise. If a few hundred steps of training match $U$'s rank agreement at similar cost, then $U$'s value reduces to the explanation it offers, the class-separability account of the paradox, rather than to a better tool. The cost-versus-$\tau$ curve settles the question. We also expect short training to be noisy for diffusion checkpoints, whose downstream utility Wu et al. found unpredictable from metrics. The closed form has no optimizer noise, and we will report the variance of each method across seeds.
Why not just condition FID on the class label? Matching each class's distribution well is not the same as keeping the classes separable in a way that transfers to real data. Two synthetic class clouds can each sit close to their real counterparts and still overlap where the real decision boundary lies. Class-conditional FID is included as an ablation, and if it matches $U$, the simpler score is the one to adopt.
Why not ask a vision-language model judge? K-Sort Eval reaches human-arena rank parity in fewer than 90 model runs by correcting general vision-language judges such as GPT-4o (K-Sort Eval). It approximates human preference, however, and here preference for realism is precisely what diverges from utility. The authors also note that the judges' visual reasoning sets a ceiling on reliability. A judge that rates realism would most likely inherit FID's sign error on StyleGAN3.
Decisive Checks
The main test is held out by generator family. We fix every design choice, including the weight rule, the feature layer and the sample count, using StyleGAN3 on AIROGS. We then report rank agreement for diffusion checkpoints on AIROGS and GOALS without further tuning. The idea fails if $U$ on RETFound features does not significantly beat RETFound-FID on these held-out trajectories, or if truncated training matches it at equal cost. The mechanistic check is whether class separability in synthetic features falls across StyleGAN3 checkpoints while FID improves. If it does not, our account of the paradox is wrong even if the proxy works. As a final consistency check, applying the same cost weighting to the prevalence-skewed AIROGS labels should change which checkpoint is chosen in the direction a clinician would want: fewer missed referable-glaucoma cases among models trained on the chosen checkpoint's output.
References
- A Pragmatic Note on Evaluating Generative Models with Fréchet Inception Distance for Retinal Image Synthesis
- Spatially adaptive multi-scale contextual attention for image inpainting
- Rethinking FID Through the Geometry of the Reference Dataset
- RaTEScore: A Metric for Radiology Report Generation
- CRG Score: A Distribution-Aware Clinical Metric for Radiology Report Generation
- Evaluating progress in automatic chest X-ray radiology report generation
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge