Generated images are now a routine way to enlarge small training sets, but many of them are wrong in a specific way. Up close they look right: the fur is sharp, the cell nuclei have plausible texture. Taken as a whole they are broken, with anatomy that could not exist, tissue architecture that makes no sense, or object parts in the wrong places. The networks used to screen these images before training are usually ordinary convolutional networks (CNNs), which judge an image by combining evidence from small neighbourhoods. Such networks struggle to see long-range structure, as Li et al. argue, so they can pass exactly these locally convincing, globally incoherent samples. We propose a cheap test that catches them. Take a classifier that can relate distant parts of an image, and measure how much its confidence drops when the image is cut into tiles and the tiles are shuffled. For a real image, shuffling destroys structure the classifier relies on, so its confidence falls a lot. For a synthetic image whose global structure was already broken, confidence barely moves, because the classifier was only ever using local texture.
The test matters for more than cleanliness. The main tension in selective augmentation is that confidence-based filters keep easy samples near class centres, which add little, while uncertainty-based filters admit noisy junk along with genuinely hard examples. A structure-dependence score can tell "hard but coherent" apart from "broken." That would let a pipeline keep informative boundary samples without poisoning the learner.
Core Claim
Predictive entropy, the uncertainty of a classifier's output, mixes up two different reasons a synthetic image can be uncertain. The image may lie near a real decision boundary, or it may be structurally corrupted. Our claim is that the two can be separated by a contrast between global and local evidence, and that only a classifier with a global receptive field can supply that contrast.
We define the score as follows. Let $p_g(y \mid x)$ be the probability that a globally connected classifier assigns to the intended class $y$ of image $x$. Let $\pi_k$ be a random permutation of a $k \times k$ grid of tiles. The structure-dependence score is
$$ S_k(x) = \log p_g(y \mid x) - \mathbb{E}_{\pi_k}\left[\log p_g(y \mid \pi_k(x))\right]. $$
A synthetic image is rejected if its $S_k$ falls below a low percentile of the $S_k$ distribution computed on real images of class $y$. Once structure-broken images have been removed, entropy or boundary-seeking selection can run on what remains.
The receptive-field argument predicts that the same score computed with a purely local CNN should be close to zero for both real and synthetic images. A network that only sees tiles is barely affected by rearranging them, so it cannot discriminate. This is a falsifiable prediction, and it isolates architecture as the working ingredient.
Relation to Prior Work
Several papers motivate the proposal by showing that indiscriminate synthesis hurts and selection helps. Pooladzandi et al. filter samples from an energy-based model by output entropy and then pick a gradient-matching subset. They report that adding all synthetic samples lowered SVHN accuracy from 92.0% to 88.1%, entropy filtering raised it to 93.8%, and the full pipeline reached 94.5%. Wang et al. keep the top 20% most confident diffusion samples per class for adversarial training at 32×32. Sampling Strategies for GAN Synthetic Data finds that selecting by the GAN discriminator's realism score (+2.6% on AffectNet) beats selecting by classifier confidence (+2.1%). That result is direct evidence that class confidence misses some realism failures.
All three score images with one number from a single network and never ask what that network can see. Our score is a contrast between two views of the image, and its usefulness is predicted to depend on receptive field.
Two further papers set the tension we target. Xue et al. (HistoGAN) keep low-entropy samples near class centroids and report +6.7% on cervical histopathology and +2.8% on a 10% PCam subset. Their generator uses self-attention, but their filter and downstream classifier are a ResNet34. TAP argues the opposite in tabular data: centroid samples are redundant, and utility lies near boundaries. We propose that part of the disagreement arises because entropy alone cannot certify that a boundary sample is coherent. Our structure gate is meant to make boundary-seeking selection safe for images.
Two papers supply the method. The global classifiers inside the filter come from ParCNetV2, which uses one-dimensional convolution kernels larger than the feature map so that every output sees the whole image. It reports 84.6% top-1 on ImageNet-1K with 56M parameters. The second source is CCNet, whose repeated row-and-column attention reaches every pixel with 11× less memory than dense non-local attention. We reuse these architectures unchanged; the contribution is the scoring rule and the analysis.
One paper warns us about a failure mode. RAID shows that AI-generated-image detectors fall to near-zero F1 under small perturbations. A learned "is this fake" detector is therefore a fragile gatekeeper. This is one reason we score whether the classifier uses structure, not whether an image is fake.
Research Program
We will build the filter in three stages. First, we train or fine-tune a global-receptive-field classifier on the real data only. The main choice is ParCNetV2, with a CCNet-augmented ResNet as a second instance. We also train a matched local ResNet whose effective receptive field we will measure explicitly.
Second, we compute $S_k$ for several grid sizes $k$, averaging over a small number of permutations; four is expected to suffice. The per-sample cost is then a handful of forward passes, far below iterative MCMC or reinforcement-learning samplers. Varying $k$ lets us ask at what spatial scale a generator's failures live.
Third, we compose the gate with existing selectors: confidence top-20%, entropy filtering, and a boundary-seeking rule that keeps high-entropy samples. The question is which of these benefit from removing structure-broken samples first.
The analysis has two strands. The first is a controlled detection study. We plant known global corruptions in generated images, such as swapped halves, duplicated object parts, and tile rearrangements, all of which preserve local statistics. We then measure how well each score separates corrupted from clean images, reported as AUROC (area under the ROC curve). The second strand tests whether the downstream model's receptive field changes how much structure-broken samples hurt. Our hypothesis is that local downstream models largely ignore such samples while global models are damaged by them. If true, this would partly explain why a ResNet34 pipeline like HistoGAN's could succeed with centroid-only selection.
Baselines
The naive reference points are real data only and all synthetic data added without filtering. Pooladzandi et al. show on SVHN that the second can fall below the first (92.0% versus 88.1%). We expect our higher-resolution settings to reproduce some version of this gap.
The strongest existing selector for natural images is Wang et al.'s per-class top-20% confidence filter. For histopathology it is HistoGAN's entropy plus feature-centroid filter, which reports +6.7% on cervical histopathology and +2.8% on PCam-10% over a ResNet34. Neither has a reported result on Lacuna for a 224-pixel ImageNet subset. For those settings we will reimplement both, using the same generator pool, the same number of admitted samples, and the same downstream training recipe.
We also include the realism-discriminator selector, cr-sam, where a discriminator is available. It is the strongest reported competitor on the realism-versus-confidence question, with +2.6% on AffectNet. We will not rerun on AffectNet, because 64×64 images blunt receptive-field effects.
Two ablations isolate the key ingredient. The first is confidence filtering with the same ParCNetV2 teacher but no shuffle contrast; it asks whether the benefit is just "a stronger teacher." The second is the shuffle score computed with the local ResNet; it asks whether the benefit is the contrast itself, independent of receptive field.
Every selector receives the same candidate pool, the same admission budget, and a comparable tuning budget, with thresholds chosen on the same held-out real validation split. We will report compute cost per candidate alongside accuracy. Beating the baselines means two things. First, the structure gate combined with any selector must beat that selector alone by more than the seed-to-seed standard deviation across at least three seeds, in at least two of the three dataset settings. Second, the gated boundary-seeking rule must match or exceed centroid selection. That second result would be the concrete evidence that the fidelity-utility dilemma is partly a filtering failure.
Why not just…?
Why not just use a stronger global model's confidence as the filter? This is the most serious alternative, and the ablation above exists to settle it. Confidence cannot distinguish an image that is uncertain because it is hard from one that is uncertain because it is broken. The AffectNet gap between cr-sam and cl-sam suggests that confidence alone leaves realism failures in the pool. If the ParCNetV2-confidence ablation matches our full method, we will report that the receptive field of the teacher, not the contrast, carries the effect. That would still be a useful and narrower finding.
Why not just run the existing ResNet filter on a heavily downsampled image? Downsampling shrinks global layout until it fits inside a small receptive field, so it could recover much of the benefit at almost no cost. We treat this as a real threat and include it as a baseline in the corruption-detection study and in end-to-end training. Our expectation is that it detects coarse layout errors but misses structure that depends on fine detail spread across the image, such as the arrangement of nuclei within tissue. Varying the grid size $k$ in $S_k$ tests exactly that. If downsampling ties with our score at every scale, we will recommend it as the simpler method.
Why not just filter with the generator's discriminator or an AI-image detector? Discriminators are tied to one generator and are unavailable for most diffusion pipelines. RAID shows that detectors trained to flag fakes are brittle, with detector AUROC falling from 0.99 to 0.67 under black-box transfer. We include the discriminator as a baseline where one exists. Our score needs only a classifier trained on the real task data.
Why not just generate far more data and skip filtering? Wang et al. needed up to 50M samples and name compute as their main limitation. Pooladzandi et al. show that unfiltered data can hurt outright. We include an equal-compute comparison: an unfiltered pool matched in total generation cost against a smaller filtered pool.
Proposed Experiments
We will use three settings, chosen because receptive field plausibly matters at their resolutions. The first is a 100-class ImageNet-1K subset at 224 pixels with 50 real images per class, augmented by class-conditional diffusion samples. The second is cervical histopathology and the 10% PCam subset, matching the HistoGAN protocol, with a GAN generator. The third is a Cityscapes-derived crop classification task, used only for the corruption-detection study, where scene layout is strongly constrained.
Downstream models are a ResNet (local) and ParCNetV2 (global), both trained on each filtered pool. The metrics are top-1 accuracy, AUROC for detecting planted corruptions, and the fraction of admitted samples that human annotators flag as structurally implausible. Annotators will judge a blinded random sample of 500 images per selector.
Three results would be decisive. The first is corruption detection: $S_k$ from the global model should reach high AUROC on planted global corruptions, while the local model's $S_k$ stays near chance. If the local model does just as well, the receptive-field thesis fails. The second is a crossed test with filter receptive field and downstream receptive field as the two factors. Admitting structure-broken samples should cost the global downstream model more accuracy than the local one. If the costs are equal, the explanation we offer for the certainty-paradox disagreement is wrong. The third is the fidelity-utility check: with the structure gate applied first, boundary-seeking selection should overtake HistoGAN-style centroid selection on the histopathology tasks. If it does not, centroid selection's advantage is not caused by hidden structural noise, and we will say so.
References
- Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining
- Generating High Fidelity Synthetic Data via Coreset selection and Entropic Regularization
- Better Diffusion Models Further Improve Adversarial Training
- Sampling Strategies for GAN Synthetic Data
- Selective Synthetic Augmentation with HistoGAN for Improved Histopathology Image Classification
- Active Tabular Augmentation via Policy-Guided Diffusion Inpainting
- ParCNetV2: Oversized Kernel with Enhanced Attention
- CCNet: Criss-Cross Attention for Semantic Segmentation
- RAID: A Dataset for Testing the Adversarial Robustness of AI-Generated Image Detectors