Transplanting Real Defects in Frozen Feature Space for Few-Shot Quality Scoring of Generated Images

Image generators now produce pictures whose textures look real but whose structure is wrong: a sixth finger, a fused wrist, a face that melts into the background. Standard quality metrics miss these errors. Distribution metrics computed on ImageNet features respond mostly to texture. Pixel metrics punish plausible detail. General multimodal chatbots rate flawed images as fine. Each new generator also brings its own failure modes, and nobody will densely annotate thousands of images every time a new model ships. We propose to build a defect detector and quality scorer from only one to five annotated examples of a new defect type. The method keeps a pre-trained image encoder frozen. It cuts the encoder's feature vectors from inside each annotated defect and pastes them into the feature maps of many flawless images, at locations that show the same kind of content. A small head trained on these synthetic examples learns to flag defective image patches. The patch-level map is then turned into a single image score by pooling that is dominated by the worst region, which is how human raters behave.

The two-sentence version: real defects are tiny and local, so the scarce examples we have should be multiplied by moving the defect itself into new contexts, not by perturbing whole images. If this works, quality metrics for generated images could track new generators with a handful of labels instead of a new dataset.

Core Claim

Our central claim is that copy-paste of real defect features inside a frozen encoder's patch-token grid is a better use of a few labeled defect examples than the alternatives. A patch token is the feature vector the encoder produces for one small square of the image, and the grid of these tokens preserves spatial layout. The alternatives are style-type feature augmentation, generator-based feature hallucination, and non-parametric novelty detection against flawless images. We expect this advantage specifically because the defects that drive human quality judgments are small and localized. We also claim that a worst-region pooling of the resulting patch map correlates better with human quality scores than average pooling. This second claim follows from the finding in AGIDefect-4K that more than 85% of defects cover under 5% of the image, and that anatomical defects lower human scores far more than background ones.

What Exists and What Does Not

Three findings motivate the proposal. AGIDefect-4K builds a 4,000-image defect dataset drawn from 15 generators. It shows that zero-shot GPT-4o and Gemini reach detection AUCs of 0.29–0.38, below chance. Its AGIDA model, a Qwen3-VL system with a frozen Segment Anything encoder, reaches 0.65 AUC and 0.800 quality SRCC, but only after full multi-task supervision on the whole dataset. Human Preference Score shows that frozen CLIP features predict human choices poorly (33.1%) until they are tuned on preference data (43.5%). It also argues that ImageNet-style features are texture-biased and blind to anatomical errors. PIPAL shows that deep metrics such as LPIPS correlate weakly with humans on GAN outputs, with SRCC around 0.49–0.51. Part of the reason is that pooled features are not shift-invariant. Our proposal takes the problem from these papers but does not adopt their solutions. Its regime is few-shot, not full supervision, and it operates on local tokens, not pooled scores.

Two lines of work supply the closest machinery, and they also constrain it. FroFA augments cached frozen ViT tokens with style operations such as brightness, contrast and posterization, and gains 4.4% absolute in 1-shot classification. It also finds that geometric transforms of the token grid hurt. That negative result is a real constraint. It warns that rearranging tokens can break what the encoder encodes, and our method must show that transplanting a local block of tokens is different from warping the grid. FroFA also only studies object categories, where a global style change keeps the label intact. For a defect, a global style perturbation may simply wash out the subtle local deviation that defines it. Tensor feature hallucination generates new spatial feature tensors from class prototypes with an MSE-trained generator. It reaches 65.07% on 1-shot miniImageNet and argues that spatial structure must be kept. We agree on keeping the spatial grid. But a generator conditioned on one or two defect examples will mostly reproduce the class mean, which is the flawless object. Transplanting the real defect tokens avoids having to learn a generator at all.

Two papers come closest in spirit. DriftAD does few-shot industrial anomaly detection on MVTec-AD, reaching 97.2% 1-shot image AUROC, by shifting frozen CLIP text embeddings according to local visual features. It trains on synthetic pixel-space pseudo-anomalies and notes a gap between those and real structural defects. Our proposal replaces synthetic anomalies with real defect features and targets generative structural errors, not surface scratches. GIQA scores generated images by density or nearest-neighbor distance in frozen feature space. Its GMM variant reaches 0.764 preference accuracy on its face benchmark. It pools globally and needs many real images. We keep its non-parametric idea as a strong baseline, applied at the patch level. UniQA reports few-label quality assessment (SRCC 0.828 on CLIVE) using a frozen vision-language backbone with a small adapter. Its target is global natural distortion, not localized generative defects. SimpleFSL warns that elaborate fusion of image and text hurts few-shot generalization. This is one reason we keep the trained head small and vision-only.

Research Program

We cache patch-token grids from two frozen encoders. The first is the Segment Anything image encoder that AGIDA uses. The second is CLIP ViT-L/14, the backbone that HPS fine-tunes. Both images and defect masks are downsampled to the token grid.

For a new defect type we have $K \in \{1, 5\}$ defective images with masks, plus a pool of flawless images from the same generator. For each defect exemplar we take the tokens under its mask, together with a one-token ring of context. We then search each flawless image for the location whose context ring is most similar, by cosine similarity, to the exemplar's ring. This places a hand defect on a hand and a face defect on a face. At that location we blend the defect tokens in:

$$z'_i = (1 - m_i)\, z_i + m_i\big(\alpha\, d_i + (1-\alpha)\, z_i\big).$$

Here $z_i$ is the flawless image's token at grid position $i$, $d_i$ is the aligned defect token, $m_i \in [0,1]$ is the softened mask, and $\alpha \sim U[0.5, 1]$ sets the strength. We add small random offsets of one token. This keeps the head from relying on exact positions, in the spirit of the shift-invariance concern raised in PIPAL.

A known failure mode is that the head learns to detect the seam of pasting, not the defect. We block this with matched seam negatives. These are transplants of flawless tokens from another flawless image, using the same masks and location rule, and they are labeled clean. A lightweight head, a two-layer convolution over the token grid, is trained to predict per-token defect probability $p_i$.

The image quality score is a soft worst-region pooling,

$$s(x) = -\tau \log \frac{1}{N}\sum_{i=1}^{N} e^{p_i/\tau},$$

where $N$ is the number of tokens and $\tau$ is a temperature. As $\tau \to 0$ this tends to minus the maximum, and as $\tau \to \infty$ it tends to minus the mean. We fit $\tau$ on the $K$ support images only. The analysis asks three questions. First, how much of the gain comes from real defect content versus seam artifacts. Second, how performance scales with $K$. Third, whether the learned $\tau$ moves toward max-pooling, as the small-area finding predicts.

Baselines

The naive reference points are two. The first is zero-shot GPT-4o and Gemini as evaluated in AGIDefect-4K, which report 0.29–0.38 detection AUC. The second is average-pooled frozen CLIP similarity with no training. Anything near these numbers is a failure.

The strongest training-free competitor is patch-level nearest-neighbor novelty scoring. This is GIQA's KNN idea moved from global features to tokens. Each test token is scored by its distance to a memory bank of tokens from the same flawless pool our method uses, and the same worst-region pooling turns the map into an image score. It uses the same encoder and the same flawless images. Its only free parameters, the neighbor count and $\tau$, get the same tuning budget, selected on the support set. There is no reported result found on Lacuna for patch-level KNN on AGIDefect-4K. The original GIQA reports 0.764 on its face benchmark, which is not comparable.

The strongest few-shot learning competitor is the same head trained on the $K$ real defect images. It uses channel-wise style augmentation from FroFA, whose classification gain is 4.4% absolute in 1-shot. It is trained with an identical step budget and architecture. A third learned competitor replaces transplanting with the conditioner–generator of tensor feature hallucination, trained on defect-type tokens from the base split.

The full-supervision ceiling is AGIDA, which reports 0.65 AUC and 0.800 SRCC on AGIDefect-4K AGIDefect-4K. We will re-score it on our held-out splits where possible, because those numbers are not computed under a held-out protocol.

Ablations isolate each ingredient. One removes the seam negatives. One transplants to random rather than context-matched locations. One transplants the exemplar's context ring without the defect core. One uses mean pooling instead of worst-region pooling.

Beating the field means the following. At $K=5$ our method must exceed both the patch-KNN and FroFA-head baselines on held-out image AUC and on SRCC with human quality scores, by more than the spread across five random support draws. Matching AGIDA's 0.65 AUC on held-out types with five examples would be a strong result.

Why not just…?

Why not just ask a multimodal LLM? The evidence is already in. AGIDefect-4K reports below-chance AUCs of 0.29–0.38, which it attributes to a systematic bias toward calling generated images flawless. We include these models as a reference point only.

Why not just do nearest-neighbor novelty detection against flawless images, with no defect labels at all? This is the real threat. It needs no augmentation, and it may catch any off-distribution token. Its weakness is the texture bias documented by HPS: a mangled hand made of realistic skin texture may sit close to real hand tokens. It also cannot tell rare-but-fine content from defects. The patch-KNN baseline settles this directly. If it matches our method, the defect labels add nothing and we would recommend the label-free method instead.

Why not just train the head on the few real defects with FroFA-style augmentation? It is simpler and has a track record. However, FroFA's style operations act on every token at once, so they preserve labels for objects but not obviously for a defect covering a few tokens. They also never place the defect in a new context. The FroFA-head baseline at identical budget answers this. The context-ring-only ablation checks whether any gain comes from the defect core or merely from extra data.

Why not just fine-tune AGIDA on the new examples? Fine-tuning a multi-billion-parameter model on five images invites overfitting, which is exactly the problem frozen-feature methods avoid. We will run a parameter-efficient fine-tune of AGIDA's quality head as an additional comparison if its weights are available. We also report it at matched wall-clock compute.

Decisive Experiments

All experiments use AGIDefect-4K under a held-out protocol. If the dataset's annotations label defect types, we hold out one defect type at a time. Otherwise we hold out one generator at a time, which models the realistic case of a new model with new artifacts. The remaining data serves only for choosing hyperparameters and training the hallucination baseline. We report image-level AUC, pixel-level AUROC against the masks, and Spearman correlation with the dataset's human quality scores, at $K = 1$ and $5$ over five support draws. As a transfer check of the score alone, we apply the pooling to AGIQA-3K's human scores, using a head adapted from five AGIDefect examples. No reported number for this setting was found on Lacuna.

Three outcomes would settle the claim. The first is whether transplanting beats patch-KNN; if it does not, the method reduces to novelty detection. The second is whether removing seam negatives inflates validation scores but lowers held-out pixel AUROC; this would show the shortcut is real and controlled. The third is whether worst-region pooling raises SRCC over mean pooling, and whether the learned $\tau$ is smaller for face and body defects than for background ones. That would tie the patch map to the human weighting reported in AGIDefect-4K.