Context Surprise: Finding Defects in Generated Images by Counting Bits

Image generators now produce pictures that look polished but still contain small, glaring mistakes: a sixth finger, a melted ear, a chair leg that merges into the floor. Humans spot these at once. The automatic scores the field relies on do not. Feature-statistics metrics such as FID respond mostly to texture and miss anatomical errors (Human Preference Score). General multimodal language models asked zero-shot to find defects score below chance, with detection AUCs around 0.29 to 0.38 on AGIDefect-4K. This is one instance of a broader gap between objective image metrics and human judgment.

We propose to score defects by how many bits a region costs to describe, using an old idea from data mining: an anomaly is whatever does not compress well under the regularities that explain everything else. The specific proposal is a two-number test. For each image region we compute its code length when the rest of the image is visible, and its code length when nothing else is visible. Both come from one masked image model trained only on real photographs. A region is flagged when seeing its surroundings makes it more expensive to encode, meaning the context actively predicts something different from what is there. In minimum-description-length terms, that is exactly the condition under which encoding the region as an exception saves bits. The test therefore comes with a threshold that needs no tuning and no labeled defects.

Core Claim

Write $L(x_R \mid c)$ for the number of bits, $-\log_2 p(x_R \mid c)$, that a masked model trained on real images needs to encode the discrete visual tokens $x_R$ of region $R$ given visible context $c$. The proposed defect score is

$$s(R) = L(x_R \mid x_{\setminus R}) - L(x_R \mid \varnothing),$$

where $x_{\setminus R}$ is every token outside $R$ and $\varnothing$ means all context is masked.

For a coherent region, context helps, so $s(R)$ is strongly negative. For a region that is simply complex or noisy, context neither helps nor hurts, so $s(R)$ sits near zero. For a structural defect, the context points confidently toward one completion, such as a five-fingered hand, while the content is another. Then $s(R) > 0$.

Subtracting the context-free code length is what separates "surprising given the scene" from "intrinsically hard to compress." This is the same distinction Minimum Description Length Clustering to Measure Meaningful Image Complexity needed to stop white noise from scoring as maximally complex. Raw code length, JPEG size, or per-patch likelihood all conflate the two.

The objective-versus-subjective gap enters through the coding space. Tokens come from a perceptually trained discrete tokenizer (an encoder that maps image patches to entries of a learned codebook, trained with a feature-space rather than pixel loss). Bits are therefore spent on visible structure rather than on sub-pixel shifts or imperceptible high-frequency detail. This is the information-theoretic analogue of the contrast-sensitivity prefilter that makes MSE perceptually sensible in PAMSE.

Relation to Prior Work

Several papers supply the method. The rule "flag a point iff encoding it as an exception shortens the total description" comes from EdgeCentric, McCatch and Automatic Parameter Selection for Non-Redundant Clustering. All three apply it to graphs, tabular data or clusters, never to image regions under a learned visual model. The white-noise result of Mahon and Lukasiewicz supplies the motivation for the subtraction term. Their method, however, ranks whole datasets by complexity, takes 2 to 8 seconds per image, and does not localize or detect anomalies. We reuse their insight, not their hierarchical clustering.

Several papers motivate and constrain the proposal. VQU-Net shows that a discrete codebook stops a network from "reconstructing its way around" anomalies. It then scores in raw pixel error, inheriting the misalignment problems; we score in bits over the discrete tokens instead. Deep Perceptual Autoencoders and PIAD replace pixel error with VGG-feature error and gain substantially: PIAD reports CIFAR-10 AUC rising from 0.609 to 0.799. DPA, however, reports that continuous perceptual autoencoders overgeneralize on diverse natural images, and both need either loss-balancing heuristics or a small labeled anomaly set for tuning. Our score has no reconstruction pathway that can overgeneralize, and its threshold is fixed at zero bits.

DO2HSC uses "compression" geometrically, confining features to a spherical shell, and says nothing about perception. It is relevant here only as a warning that distance-based latent scores have blind spots.

On the evaluation side, AGIDefect-4K reports that over 85% of defects cover less than 5% of the image. This argues for local scores over global pooling, and is the main reason we score windows rather than images.

Research Program

The model is a masked token predictor over a frozen perceptual tokenizer, trained only on real photographs. Generated images are never used in training. This choice matters. A context model trained on generator outputs could share the generators' blind spots and assign defects high probability.

For each sliding window of $k \times k$ tokens we compute both code lengths with the same network. The joint code length over the window uses an autoregressive ordering inside the masked window. Batching many masks per forward pass keeps cost to a few passes per image.

The outputs are a defect map, $\max(s, 0)$ per token. They also include an image-level score: the total bits saved by encoding all positive-score windows as exceptions, which is a proper two-part MDL quantity rather than an ad hoc sum.

The analysis has three parts. First, a controlled synthetic study on real photographs. We paste in three kinds of patch: patches from unrelated images (structurally incoherent), white noise, and high-texture patches from the same scene. Only the first kind should give $s > 0$, whereas raw conditional bits should fire on all three. Second, we vary the tokenizer's rate and its training loss (perceptual versus pixel) to test whether perceptually aligned coding is what makes the score track human judgments. Third, we measure stability under imperceptible perturbations. A score built on discrete tokens and differences of code lengths should move little when noise does not flip tokens. We test this rather than assume it, contrasting with the adversarial fragility documented for neural quality metrics in BiRQA.

Baselines

The naive references are chance (AUC 0.5) and the global file size of the image under a standard lossless or JPEG coder. The latter tests whether "compression" alone, without context or perception, carries any signal. Both are run on the identical AGIDefect-4K test images at native resolution.

The key ablation is raw conditional code length $L(x_R \mid x_{\setminus R})$ without the context-free subtraction. It uses the same model, the same windows and the same compute, so it isolates the proposal's central ingredient. If the subtraction does not raise detection and localization AUC over this ablation, the core claim fails. A second ablation swaps the perceptual tokenizer for one trained with pixel loss at matched codebook size and rate, isolating the perceptual-alignment ingredient.

The strongest label-free competitor is a perceptual reconstruction detector in the style of DPA. We train it on the same real-photo corpus with a matched parameter count and take its per-location VGG-feature error as the defect map. Its hyperparameters get the same small tuning budget we spend on window size $k$. If we use any labeled anomalies for that tuning, both methods get the same 0.5% anchor set. Beating it means higher detection and localization AUC at matched compute.

The strongest reported system is AGIDA from AGIDefect-4K, a fine-tuned Qwen3-VL with a SAM encoder. It reports detection AUC 0.65 and quality SRCC 0.800, while zero-shot GPT-4o and Gemini cluster at 0.29 to 0.38 AUC. AGIDA is supervised on defect annotations. We compare against its reported number on the standard split. Matching 0.65 with zero defect labels would already be a meaningful result; exceeding it would be a strong one. For localization, no reported result was found on Lacuna for pixel-level defect localization on AGIDefect-4K, so we report our own runs of all baselines.

Why not just…?

Why not just use per-patch likelihood or a standard compressor? These measure how complex a region is, not whether it fits. Mahon and Lukasiewicz show that Shannon entropy, Kolmogorov approximations and JPEG ratios all rank white noise as maximally complex. In generated images, the analogue is that hair, foliage and fabric would dominate the map. Our raw-conditional-bits ablation and the pasted-texture control directly test whether the subtraction term fixes this.

Why not just use a perceptual autoencoder's reconstruction error? It is the natural label-free choice, and PIAD's gain over MSE shows perceptual scoring helps. DPA, however, reports overgeneralization on diverse natural images, which are exactly the domain of text-to-image output. A defect that looks locally plausible, like a well-rendered extra finger, may reconstruct well because it is not unusual as a texture, only as a structure. The matched DPA-style baseline will show whether this concern is real on AGIDefect-4K. If the autoencoder matches us, the honest conclusion is that the coding view adds interpretability and a tuning-free threshold but not accuracy.

Why not just train a supervised defect segmenter on AGIDefect-4K? This is the real threat. With 3,000 annotated defective images, a supervised segmenter may well beat a label-free score on the in-distribution split. Our claim is about generalization and supervision cost, not in-distribution accuracy. We therefore train such a segmenter on the train split and evaluate it under a leave-generators-out protocol: train on images from some of the 15 generators, test on the rest. Defect styles shift as generators improve. If the supervised model still wins on held-out generators, we report that plainly. We then test whether adding $s$ as an input feature improves it, which would reframe the contribution as a complementary signal.

Proposed Experiments

The primary benchmark is AGIDefect-4K, scored three ways. Image-level detection AUC uses the image-level MDL score, with zero labels and the threshold fixed at zero bits. Pixel-level localization AUC and IoU against the provided masks use the token defect map. We also report per-generator AUC under the leave-generators-out split, which tests whether models related to our tokenizer's training data are systematically harder.

Secondary evaluation uses EvalMuse-40K. We correlate the image-level score with human ratings restricted to images carrying structural-hallucination tags. Our score ignores the text prompt, so it is not comparable to FGA-BLIP2's prompt-alignment SRCC of 0.7742. We report its correlation on the structure subset and its gain when fused with FGA-BLIP2. No reported result was found on Lacuna for a structure-only subset of EvalMuse-40K.

The perturbation study applies small gradient-free and transfer perturbations to defect-free images. It reports how much the image score changes against the same perturbations' effect on a learned quality metric, using images from RAID as an additional source of adversarially processed generated images.

Three decisive checks would make the argument. First, on the synthetic control, $s$ separates pasted incoherent patches from noise and texture patches while raw bits do not. Second, on AGIDefect-4K, the full score beats the raw-bits ablation and the matched perceptual autoencoder on both detection and localization, and approaches or exceeds AGIDA's 0.65 detection AUC with no defect labels. Third, under leave-generators-out, the label-free score degrades less than the supervised segmenter.

Failure on the first check would mean the context-free subtraction does not separate incoherence from complexity, and the idea should be dropped. Success on the first and failure on the second would mean the principle is sound but current token models are too weak to see anatomy, which would point to scaling the context model as the next step.