Spectral diversity metrics such as the Vendi Score count the "effective number of distinct items" in a set of generated images. They do this by embedding every image with a pretrained network, building a matrix of pairwise similarities, and computing the entropy of that matrix's eigenvalues. These metrics are now used both to grade generators and to steer them toward more varied outputs. They have a known blind spot: anything that makes two embeddings point in different directions counts as diversity. That includes the things image-quality researchers have spent a decade documenting, namely ringing, aliasing, GAN texture noise, and mangled hands. As a result, a generator that adds a different small defect to every image can look more diverse than one that renders the same content cleanly. Diversity guidance can then reward the generator for drifting into damaged regions.
This note proposes to measure that inflation directly and then remove it. The measurement uses human-calibrated image-quality data, where we know two images show the same content and differ only by distortion. The fix is a similarity function trained to ignore perceptual damage while still separating content. The question is not "how do we add a quality term to diversity," which existing work already does with a scalar product. The question is "how much of the diversity we currently report is actually damage, and can the similarity function itself be made to stop counting it?"
Core Claim
Our hypothesis is that, under off-the-shelf embeddings (CLIP, DINOv2, Inception), Vendi Scores of image sets are measurably inflated by distortions and generative defects that humans regard as damage rather than variety. We further hypothesize that a similarity function made invariant to such damage, using distortion pairs with human ratings, will remove most of this inflation without collapsing genuine content diversity.
Concretely, write $\mathrm{VS}_k(X)$ for the Vendi Score of a set $X$ of $N$ images under similarity function $k$: the exponential of the entropy of the eigenvalues of $K/N$, where $K_{ij}=k(x_i,x_j)$ (The Vendi Score). For a set built from $r$ distinct contents, each shown under $m$ distortions, an ideal perceptual diversity metric should return roughly $r$, not something approaching $rm$. We define the inflation ratio
$$\rho_k = \frac{\mathrm{VS}_k(X_{\text{distorted}})}{\mathrm{VS}_k(X_{\text{clean}})},$$
where $X_{\text{clean}}$ contains the $r$ undistorted references and $X_{\text{distorted}}$ contains their degraded versions. A value of $\rho_k=1$ means damage adds no measured diversity.
The new contributions are this inflation measurement, built on existing human-rated data, and a learned damage-invariant similarity function that drives $\rho_k$ toward 1. The constraint is that agreement with human diversity judgments must be kept or improved.
Relation to Prior Work
The motivating evidence comes from both directions. The original Vendi paper (The Vendi Score) shows that random noise scores near the ceiling, which means the metric is blind to quality. Contextualized Vendi Score Guidance steers diffusion sampling with a differentiable Vendi Score on CLIP features. Its authors found that unconstrained guidance produces artifacts, and they patched this with curated real exemplar images. Benchmarking Diversity via Attribute-Conditional Human Evaluation reports that the Vendi Score on ViT embeddings conflates stylistic or textural variation with semantic diversity.
On the perceptual side, PIPAL shows that deep metrics are fragile to GAN-style distortions and spatial misalignment. Human Preference Score argues that CLIP-like features miss structural failures. AGIDefect-4K finds that most generative defects are small and localized. None of these works quantify what distortion does to an eigenvalue spectrum, and that is the measurement we add.
Several works combine quality with spectral diversity but do so differently. Quality-Weighted Vendi Scores multiplies mean scalar quality by diversity. The diversity term itself still counts defects as variety, so a defective-but-varied set can trade quality loss for diversity gain. Diversity Preserving Coresets for IQA fuses a quality-aware and a content-aware distance matrix before spectral selection. It deliberately counts distortion variety as diversity, because its goal is training IQA models. We want the opposite: distortion variety should be excluded, so their quality-aware view becomes our nuisance to suppress. Conditional Vendi Score removes prompt-induced variation from the score. We remove damage-induced variation, and the two corrections are complementary. DRAGON optimizes diffusion models on batch rewards including Vendi, mainly in audio. It inherits whatever the embedding counts as diversity.
Perceptual Fidelity Aware MSE supplies one method we reuse as a baseline: low-pass filtering before comparison, to mimic human contrast sensitivity.
Research Program
The first stage is the measurement. From PIPAL's 250 references and 40 distortion types, we build sets of $r$ references times $m$ distortions. We stratify by distortion family (traditional, restoration, GAN-based) and by human-rated quality level. We then compute $\rho_k$ for CLIP, DINOv2, Inception, LPIPS-derived similarities, anti-aliased (BlurPool) feature extractors, and low-pass-before-embedding variants. KADID-10K and TID2013 serve as secondary sources of distortion pairs.
PIPAL distortions are restoration artifacts, not generative ones. We therefore also build a generative counterpart from AGIDefect-4K's pixel-level defect masks. For each defective image, we inpaint the masked region with the same generator to produce a repaired twin that differs only locally. Inflation is then the ratio of Vendi on defective sets to Vendi on their repaired twins. A first honest outcome is possible here. Because over 85% of these defects cover less than 5% of the image, global embeddings may barely move, and inflation from generative defects may be small. That would itself be a useful negative result. It would shift the proposal's weight toward restoration and texture artifacts.
The second stage is the fix. We train a small projection head on frozen DINOv2 features with a contrastive objective. Positive pairs are the same PIPAL or KADID reference under different distortions. Negatives are different references. We add a regularizer that preserves the spectrum of clean-reference sets, which guards against the trivial solution of making everything similar. Defective/repaired twins from a training split of AGIDefect-4K are added as further positives. The resulting similarity function $k_\theta$ defines a damage-invariant Vendi Score. We also report Hill-number orders from Cousins of the Vendi Score, because higher orders weight dominant eigenvalues and may react to inflation differently.
The third stage tests consequences. We swap $k_\theta$ into Vendi-guided diffusion sampling in the style of c-VSG and Conditional Vendi guidance, with no exemplar bank. We then ask whether defect rates fall while measured diversity, judged by humans and by a held-out embedding, is preserved.
Baselines
The naive reference point is the plain Vendi Score on CLIP and on ViT/DINOv2 embeddings, exactly as used in The Vendi Score and the attribute-conditional benchmark. For agreement with human diversity judgment, the attribute-conditional benchmark reports about 80% alignment for Vendi on ViT embeddings in cases with clear diversity gaps (Benchmarking Diversity). It also reports 92.6% accuracy for Gemini 2.5 Flash with structured counting prompts on its Golden Set. Beating plain Vendi means lower $\rho$ at equal or higher human agreement on the same 86 concept-attribute pairs. We do not claim to beat the Gemini judge, which is far more expensive. We report it as an upper reference.
The strongest metric-level competitor is filter-then-measure. We drop images flagged by a defect detector and compute Vendi on the survivors. The detector is AGIDA from AGIDefect-4K, which reports a detection AUC of 0.65, alongside an IQA quality model thresholded at matched retention rates. The second is quality-weighted Vendi (Quality-Weighted Vendi Scores), using the same quality model. Both receive the same backbone and an equal threshold or weight tuning budget on the same validation split.
The strongest guidance-level competitor is c-VSG with its exemplar bank (Contextualized Vendi Score Guidance). It reports a 40% relative improvement in worst-region F1 on GeoDE. No reported result was found on Lacuna for c-VSG's defect rate. We will run it with the same generator, step count and guidance strength sweep. Beating it means matching its worst-region F1 without an exemplar bank while showing a lower defect rate.
Two ablations isolate the key ingredient. The first trains the same projection head with the same contrastive objective but on generic augmentations (crops, color jitter) instead of human-rated distortions. This tests whether perceptual distortion data matters or whether any invariance training suffices. The second uses only the low-pass or BlurPool variant with no learning.
Why not just…?
Why not just filter out bad images, then measure diversity? Filtering removes whole images, so a set with mild, ubiquitous texture noise either loses most of its members or keeps the inflation. It also depends on a detector that, per AGIDefect-4K, reaches only 0.65 AUC on generative defects. Even so, this is a genuine threat on the generative-defect side. If filtering at matched retention yields $\rho$ as low as $k_\theta$ on the defective/repaired twins, we will report that filtering suffices for localized defects. In that case we confine our claim to distributed artifacts, where filtering cannot help without discarding nearly everything.
Why not just use quality-weighted Vendi? Multiplying by mean quality lowers the score of damaged sets, but it leaves the diversity factor inflated. A generator can therefore offset quality loss with artifact-driven diversity, and guidance gradients still point toward varied damage. The inflation measurement shows this directly: qVS's diversity factor has the same $\rho$ as plain Vendi. The guidance experiment tests whether that matters in practice.
Why not just blur images or use an anti-aliased backbone before embedding? This is the cheapest fix. It is grounded in the contrast-sensitivity argument of PAMSE and the BlurPool gain on PIPAL's GAN subset, where LPIPS SRCC rose from 0.4862 to 0.5135 (PIPAL). It is a real threat for high-frequency artifacts, and it is a named ablation. We expect it to fail on structural defects and low-frequency distortions such as color shifts and warping, and to erase legitimate fine-texture diversity. Our evidence for that would be lower human agreement on texture-attribute pairs in the benchmark. If it matches $k_\theta$ everywhere, the learned head is unnecessary and we will say so.
Proposed Experiments
The first decisive check is the PIPAL inflation table. We report $\rho$ per embedding and per distortion family, with the target that $k_\theta$ achieves $\rho$ near 1 on held-out distortion types. This requires holding out entire distortion families, not just images, so that we test generalization rather than memorization. A clean-reference control must accompany it: on sets of distinct references only, $\mathrm{VS}_{k_\theta}$ must remain near $r$. Otherwise the reduction in inflation comes from collapse.
The second check uses AGIDefect-4K defective/repaired twins on a held-out generator split. We report $\rho$ and compare against filter-then-measure and qVS at matched settings.
The third check uses the attribute-conditional benchmark. The new metric must retain or exceed roughly 80% human agreement. We report this separately for texture-type attributes, where damage-invariance risks hurting.
The fourth check is guidance with Stable Diffusion on GeoDE and DollarStreet, comparing CLIP-kernel Vendi guidance, c-VSG with exemplars, and $k_\theta$ guidance without exemplars. Each method is scored on worst-region F1, on defect rate measured by AGIDA and by human raters on a sample, and on HPS. Each method is also scored by a Vendi Score computed with a different backbone than the one used for guidance, so that no method is graded by its own similarity function.
The proposal succeeds if three things hold together: measurable inflation exists under standard embeddings, $k_\theta$ removes it on held-out distortions without collapse, and guidance with $k_\theta$ lowers defect rates relative to CLIP-kernel guidance at comparable worst-region F1. If inflation turns out to be negligible for generative defects, the measurement study alone settles an assumption the field currently makes without evidence.
References
- The Vendi Score: A Diversity Evaluation Metric for Machine Learning
- Improving Geo-Diversity of Generated Images with Contextualized Vendi Score Guidance
- Benchmarking Diversity in Image Generation via Attribute-Conditional Human Evaluation
- PIPAL: A Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference
- AGIDefect-4K: A Richly Annotated Dataset for AI-Generated Image Defect Detection, Localization and Explanation
- Quality-Weighted Vendi Scores And Their Application To Diverse Experimental Design
- Diversity Preserving Coresets for Image Quality Assessment
- Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs
- DRAGON: Distributional Rewards Optimize Diffusion Generative Models
- Perceptual Fidelity Aware Mean Squared Error
- Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning