Spending the Synthetic Image Budget Where Vision Models Agree

Synthetic image-text data is expensive to make. Every training pair costs a full run of a diffusion model (a generator that turns random noise into an image over dozens of denoising steps), and most pipelines then add further passes through large models to check that the image matches its caption. This proposal tests a cheaper signal for deciding which candidate images deserve that expense. The signal is whether several small, independently trained vision models "see" the image the same way. The signal is free of text and needs no labels, and it can be computed on a half-finished image. We use it to cut generation short on candidates the vision models disagree about, and to pick which finished pairs a small vision-language student is trained on.

The motivating finding comes from Modulating Cross-Modal Convergence with Single-Stimulus, Intra-Modal Dispersion. That paper measured, for each image, how far four vision transformers' representations scatter around their shared consensus. It found that images with low scatter align with language-model representations of their captions by up to a factor of two more than high-scatter images. The paper used this only as a diagnostic on fixed models. We turn it into a data-allocation rule, and we add a check that paper did not run. Filtering toward images that models agree on could strip out the "messy" variety that Real-World Unsupervised Models Generalize to Predict Brain Responses to Out-of-Distribution Stimuli found makes self-supervised models more brain-like. We therefore measure the student's alignment with primate and human visual cortex alongside its machine benchmarks.

Core Claim

Cross-model dispersion, scored by a panel of small self-supervised vision encoders, is a proxy that meets three conditions. It predicts how well a synthetic image will support image-text alignment. It can be estimated early in diffusion sampling, which allows early abort. It adds information beyond both text-image similarity scoring and single-model typicality.

The concrete test is a small CLIP-style student trained from scratch on a fixed compute budget. If the claim holds, a student trained on dispersion-selected data should reach a smaller modality gap and higher zero-shot accuracy than a student trained on randomly subsampled synthetic data of the same size. It should get there using less generation compute than the unfiltered set requires. Its brain alignment should not fall below the random-subset student. If brain alignment does fall, that result is itself informative: it would show that what makes data good for machine cross-modal alignment trades off against what makes it good for brain alignment.

Research Program

Scoring panel. The panel contains $K$ frozen, purely self-supervised vision encoders. Candidates are an MAE and a DINO ViT, a SimCLR ResNet-18, and an Instance-Prototype Contrastive Learning model from A self-supervised domain-general learning framework for human ventral stream representation. CLIP is deliberately excluded, so the score never sees text and cannot simply reproduce the student's own objective.

On a fixed reference set of real images, we reduce each encoder's normalized embeddings $\phi_k(x)$ to a common dimension. Following the adapted Generalized Procrustes Analysis in the dispersion paper, we then fit orthogonal maps $R_k$ that rotate every encoder into a shared consensus space. The dispersion of a new image $x$ is

$$d(x) = \frac{1}{K}\sum_{k=1}^{K} \big\| R_k \phi_k(x) - \bar z(x) \big\|^2, \qquad \bar z(x) = \frac{1}{K}\sum_{k=1}^{K} R_k \phi_k(x),$$

that is, the mean squared distance of the $K$ rotated embeddings from their average. Scoring costs $K$ forward passes of small encoders. That is a small fraction of one diffusion sample.

Generation with early abort. A language model writes captions, and a text-to-image diffusion model renders each caption. The caption is the text side of the pair. At an intermediate step, for example a quarter of the way through sampling, we take the denoiser's current estimate of the clean image and score its dispersion. Candidates in the high-dispersion tail of their batch are abandoned. The remaining candidates finish sampling and are scored again. Early abort only makes sense if early-step dispersion ranks candidates much like final-step dispersion does. We measure this rank correlation first. If it is weak, the early-abort component is dropped and the method reduces to pure curation.

Selection policy. Choosing only the lowest-dispersion images may favor prototypical, low-variety images. Beyond Quantity: Trajectory Diversity Scaling for Code Agents shows that low-diversity synthetic data plateaus. The adversity result points the same way. We therefore compare selection by dispersion band (lowest, middle, highest) against a stratified rule that removes only the extreme high-dispersion tail. We expect the tail-removal or middle-band rule to be best. The experiment decides between them.

Per-stimulus biological check. Before training anything, we compute $d(x)$ for the stimulus images in the Brain-Score suite (Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?) and for the 72-image Inanimate Objects fMRI set. We then ask whether low-dispersion images are better predicted by existing models' neural encoding fits. The dispersion paper raised this biological question but left it untested. The answer also tells us in advance whether dispersion filtering should help or hurt brain alignment.

Baselines

The naive reference point is a student trained on a uniformly random subset of the synthetic pool. The subset matches the filtered set in size and training steps, so any gain from filtering cannot come from seeing fewer or more examples. A second naive reference trains on the entire unfiltered pool. It gives an upper bound on what extra data buys, and it gives the compute denominator for the overhead comparison. All students share architecture, optimizer, a hyperparameter search of equal trial budget, and total training tokens.

The strongest filtering competitor is text-image similarity filtering: keep the pairs whose image and caption score highest under a pretrained CLIP. It is cheap, standard, and aimed directly at the pairing quality the student is trained on. It will be run on the same candidate pool at the same retention rate, and also at the same early-abort step using the partial image. No reported result was found on Lacuna for similarity-score filtering of synthetic image-text pairs, so its numbers come entirely from our runs. Beating it means higher zero-shot accuracy and a smaller modality gap at equal retained size. Failing that, the combined filter (dispersion plus similarity) must beat similarity alone. If neither holds, dispersion adds nothing and the claim fails.

The strongest architectural competitor is AlignCLIP from Mitigate the Gap: Improving Cross-Modal Alignment in CLIP. It shares transformer and projection layers across modalities and adds sentence-encoder-guided separation within each modality. Its authors report that it raised average positive-pair cosine similarity from 0.47 to 0.67 and improved zero-shot top-1 accuracy by about 8% on CIFAR-10 and CIFAR-100, trained on the full CC12M. We treat it as orthogonal rather than rival. We train AlignCLIP-style students on random and dispersion-filtered synthetic data under the same budget. Data selection should add to the architectural gain, not replace it.

For the brain comparison, the reference point is the best self-supervised model trained on real uncurated video. Real-World Unsupervised Models Generalize to Predict Brain Responses to Out-of-Distribution Stimuli reports that such models beat the previous Brain-Score leader by at least 47%. We do not expect a small synthetic-data student to match this. The question is only whether dispersion filtering moves brain scores toward or away from it, relative to the random-subset student.

The ablation that isolates the key ingredient replaces cross-model dispersion with single-model typicality: the distance from an image's DINO embedding to its nearest reference neighbors. If typicality works as well, the effect is about choosing "ordinary-looking" images, and agreement across models adds nothing. A second ablation keeps the filter but removes early abort. This separates the compute saving from the curation effect.

Why not just…?

Why not just filter by CLIP image-text similarity? This is the most serious threat, and it may well match dispersion on zero-shot accuracy. Two considerations make dispersion worth testing anyway. First, similarity filtering uses a model trained with the same contrastive objective, and Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning shows that objective shapes a gap whose size carries both accuracy and fairness consequences. Filtering with that model risks copying its geometry into the student. Second, dispersion is text-free and can be computed without a caption. The head-to-head and combined-filter runs, with FairFace misclassification rates reported alongside accuracy, decide whether dispersion adds information or should be abandoned.

Why not just generate fewer images and skip filtering? The random-subset baseline is exactly this. If it matches the filtered student at equal compute, the proposal fails on its central claim. The early-abort accounting reports GPU-hours spent on aborted samples. The saving must therefore be net of wasted partial generations.

Why not just train on real uncurated streams instead of synthetic data? The adversity result favors real data for brain alignment, and we take that seriously enough to include it as the brain reference point. Synthetic generation remains useful where paired text is missing. From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data shows synthetic pairs, including negative captions, fixing failures that passive data leaves in place: a 13% F1 gain in hallucination detection over positive-only training. Our aim is to make synthetic data cheaper, not to argue it beats real video on brain fit.

Why not just fix the gap after training, with a mean shift or a closed-form linear alignment? Subtracting the mean offset between modalities distorts neighborhoods within each modality (Mitigate the Gap: Improving Cross-Modal Alignment in CLIP). The closed-form singular value decomposition solution in Towards Achieving Perfect Multimodal Alignment cut alignment error from 2210 to 8.66 on the UTD Multi-Modal Human Action Dataset (UTD-MHAD). Its authors state, however, that it is linear, becomes noise-sensitive when samples outnumber dimensions, and struggles with ambiguous language. We apply it post hoc to every student anyway. If it erases the difference between filtered and random students, the benefit of filtering was only a correctable offset and not better semantic structure.

Proposed Experiments

The candidate pool is roughly a million generated pairs, filtered at several retention rates. Machine evaluation covers zero-shot top-1 accuracy on CIFAR-10, CIFAR-100 and ImageNet-1K, and robustness on ImageNet-R and ImageNet-Sketch. It also covers retrieval on MSCOCO and Flickr30K, together with positive-pair cosine similarity and the gap distance used by AlignCLIP. Local cross-modal alignment is measured with Centered Kernel k-Nearest Neighbors Alignment on the 1,024 Wikipedia caption pairs from the dispersion paper. Brain evaluation applies Brain-Score V4, IT, and behavioral predictivity to each student's vision tower. We also run voxel-wise encoding similarity analysis on the Object Orientation and Inanimate Objects fMRI sets, with anterior occipitotemporal cortex as the region of interest, since models explain only about 32% of its variance there. We add correlation with human error patterns on CIFAR-10H, where Exploring internal representations of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects found self-supervised contrastive models reaching Spearman correlations near 0.8 to 0.9. Cost is reported as total generation and scoring GPU-hours per retained pair.

Four checks are decisive. First, early-step and final-step dispersion must rank candidates consistently, or early abort is dropped. Second, at equal retained size, some dispersion policy must beat random selection on zero-shot accuracy and gap metrics. It must also beat or complement similarity filtering, and it must beat single-model typicality. Third, the gain must survive post-hoc linear alignment. Fourth, the brain-score change relative to the random student, together with the per-stimulus analysis on Brain-Score images, tells us whether "vision models agree" tracks what the cortex represents well or trades against it. Either outcome answers a question the dispersion paper left open.