Fuzzing Vision-Language Models with Evolving Scene Programs

Vision-language models such as CLIP embed images and captions in a shared vector space. They still fail at telling "a red apple and a green book" from "a green apple and a red book." The benchmarks that expose these failures are small and made by hand. Winoground has 400 curated photo pairs. ColorSwap needed diffusion-generated images plus human filtering, because the image generators make the same binding mistakes as the models under test. Software testing has solved a similar problem. Instead of asking a language model to emit a hard-to-get-right input directly, you ask it to write a small program that builds the input, and then you evolve that program whenever testing stops finding anything new. We propose to apply this idea to vision-language models. A language model writes scene programs, meaning short Python scripts that place objects with chosen attributes and relations and render them. The caption comes from the scene description by a fixed template, so image and caption agree by construction. When the programs stop producing new model failures, the language model mutates the programs themselves to explore new kinds of scenes.

The payoff is a failure-finding loop that runs without human labels, is correct by construction, and produces an open-ended set of paired minimal tests. It also gives an experimental lever that the modality-gap literature has lacked. With a controlled stream of compositional minimal pairs, we can ask whether the geometric separation between image and text embeddings has anything to do with compositional failure. We can also test whether training on a broad, automatically discovered set of failure types transfers to natural-photo benchmarks where narrow synthetic fine-tuning has not.

Core Claim

We claim two things. First, mutating the generator program, guided by a coverage signal defined over the scene programs, finds more distinct compositional failure types per unit of compute than randomly sweeping the parameters of a fixed, hand-designed scene grammar. Second, fine-tuning a dual-encoder model (separate image and text encoders, as in CLIP) on the union of discovered failure types transfers to Winoground and EqBen in a way that single-skill synthetic fine-tuning does not. Both claims can fail. If a well-designed fixed grammar with random sampling matches the evolving generator, the first claim is false and the language-model mutation step is unnecessary.

Where This Sits

The mechanism comes from G²FUZZ, which has a language model write Python scripts that call libraries like PIL and pypdf to produce TIFF, MP4 and PDF files. It repairs those scripts using error tracebacks, lets the AFL++ fuzzer mutate the output bytes, and asks the model to rewrite the script whenever code coverage plateaus. With this approach it found over 15,000 more code edges than AFL++, FormatFuzzer and Fuzztruction on UNIFUZZ. We keep the structure: program synthesis, execution feedback, and program-level mutation at plateaus. We change the target and the reward. The target is a neural model rather than a C parser. The reward is misranked minimal pairs, and coverage becomes "which kinds of scene have produced failures" instead of which code branches were reached. Parameterized generator classes in Themis and HardTests supply the practice of seeded, parameterized generation that each scene program follows.

Three lines of work motivate the proposal and also set limits on it. Winoground and ColorSwap show that dual encoders sit near chance on minimal pairs. On ColorSwap's group score, CLIP reaches 12% and SigLIP 30%. ColorSwap also delivers the key negative result. Fine-tuning CLIP on 1,400 color-swap pairs lifts ColorSwap from 11.67% to 63% but does not carry over to Winoground. The authors attribute this to compositional skills that are learned separately. Our second claim tests whether broad, automatically discovered coverage fixes that, or whether it simply repeats the failure at larger scale.

Why is Winoground Hard? finds that CLIP's text side alone separates the paired captions with over 80% accuracy. The failure happens where the image and text representations are matched. Success falls to 0% on small or blurry objects, and unusual images hurt further. This supports generating visual counterfactuals rather than textual ones. It also warns about a real risk: rendered scenes are themselves unusual images, so transfer to photographs cannot be assumed.

The geometry papers make up the third line. Mind the Gap shows that image and text embeddings sit in separate cones and that widening the gap can help zero-shot accuracy. Closing the Modality Gap Aligns Group-Wise Semantics (ModGap) and Mitigate the Gap (AlignCLIP) argue the opposite: that closing the gap improves clustering and robustness. None of the three measures fine-grained compositional minimal pairs. Our controlled failure stream lets us test their disagreement on exactly that axis.

Research Program

Scene programs. Each scene program is a Python function. It samples a scene graph (a list of objects with attributes plus relations between them) and renders it with a 3D engine using CLEVR-like primitives extended with textured everyday assets. The CLEVR line (CLEVR, CLOSURE) shows such rendering is dependable. Here the grammar is not fixed but written and rewritten by a language model.

Minimal pairs. A program yields a pair of scene graphs that differ by one permutation of the caption's parts, such as swapping attributes, swapping relation arguments, or swapping counts. The two captions are filled from the scene graphs by a deterministic template, so they use the same words in a different arrangement.

Invariant checks. Before a pair counts as a test, execution checks it. Each named object must cover a minimum number of pixels in its segmentation mask. Relations are verified from rendered 3D coordinates. Both images must differ only in the intended property. These checks play the role that parsers and compilers play in G²FUZZ. Tracebacks and failed checks go back to the language model for repair.

Failure signal. For the model under test, a pair fails if the Winoground group criterion is violated. The group criterion requires each image to prefer its own caption and each caption to prefer its own image.

Coverage. Coverage is the set of "failure signatures." A signature is the abstract type of the swap together with the scene-graph features involved. One example is attribute swap × material × two objects × occlusion present. When the discovery rate of new failing signatures plateaus, the language model receives the current coverage map and a sample of successful and failing programs. It is asked to write a program that exercises signatures not yet covered. This is the step that separates the method from a random parameter sweep.

Geometry analysis. For every tested pair we log the local modality gap, following the distance-between-centroids measure of Mind the Gap, together with the margin between correct and swapped scores. The same loop is then run against CLIP, SigLIP, a ModGap-trained model and an AlignCLIP-style model. This produces a per-signature map of where each geometry fails.

Training. We fine-tune CLIP with hard negatives drawn from discovered pairs, matched in size to the ColorSwap fine-tuning set and at larger sizes. This tests whether gains from broad coverage transfer.

Baselines

The naive reference is the frozen model with no intervention. It is scored on Winoground, ColorSwap and EqBen, and on the same generated test pool. The reported anchors are CLIP 12% and SigLIP 30% group score on ColorSwap (ColorSwap), and 14.50% for VinVL and 14.25% for FLAVA on Winoground against 16.67% chance (Winoground). We rerun CLIP on Winoground ourselves, since the file gives no CLIP Winoground group score.

The ablation that matters most is a fixed hand-written generator over the same rendering engine and object vocabulary. It has a grammar a careful researcher would write, covering colors, materials, sizes, spatial prepositions and counts, and it samples parameters uniformly at random. This is the natural CLEVR-style competitor, and it isolates the value of language-model mutation of programs. It gets the same number of rendered pairs and the same GPU hours, and its grammar is written before any failure data is seen. A second ablation keeps language-model program synthesis but removes the coverage feedback, so the model simply writes new programs without seeing the coverage map. This separates "a language model writes diverse scenes" from "coverage steers the language model." The evolving generator beats both if it finds significantly more distinct failing signatures at equal compute, and if its discovered pairs, used for fine-tuning, lead to higher Winoground and EqBen group scores.

For discovery, the strongest existing pipeline is ColorSwap's diffusion-plus-human-filter construction. We compare against its reported cost structure and its label-error rate. As an automated variant, we also run Stable Diffusion with automatic filtering by a strong non-contrastive scorer. The file reports BLIP image-text matching at 87% group score on ColorSwap, so BLIP is the natural filter. Beating this variant means a higher human-audited label-validity rate and more failure signatures per compute budget.

For downstream improvement, the strongest baseline is CECE (Natural Language Inference Improves Compositionality). CECE expands captions at test time into statements the caption implies and statements it rules out. It reports +19.2% Winoground group score over the prior best and +12.9% on EqBen, but the file gives no absolute value. We run CECE on the same CLIP backbone with the same scoring budget. Our fine-tuned model counts as beating it if it matches or exceeds CECE's Winoground group score without CECE's extra test-time language-model calls. It also counts if it is complementary to CECE, meaning that stacking the two beats CECE alone.

The ColorSwap fine-tuning recipe, 1,400 pairs lifting CLIP from 11.67% to 63% on ColorSwap, is the narrow-coverage training baseline. We match its data size. For ModGap and AlignCLIP we found no reported result on Lacuna on Winoground or ColorSwap, so we measure these ourselves.

Why not just…?

Why not just write a good procedural generator by hand, as CLEVR did? This is the real threat. The fixed-grammar ablation exists to settle it. A hand-written grammar can only produce failures along the axes its designer thought of. The claim is that the language model, shown the coverage map, proposes new axes such as partial occlusion, reflections, relative size inverting an apparent relation, or text rendered inside the scene. If the fixed generator finds as many distinct failing signatures and yields equal transfer, we will report that the language-model step adds nothing. The contribution would then reduce to the geometry analysis and the broad-coverage training study.

Why not just generate images with a diffusion model and filter automatically? ColorSwap found that Midjourney, DALL-E 3 and Stable Diffusion mis-bind attributes badly enough that humans had to filter. Any automatic filter strong enough to replace them would itself have to solve the binding problem we are probing. The automated diffusion baseline measures this directly through audited label error. Diffusion does keep one real advantage: it produces natural-looking images. That is why the transfer experiments also test whether rendered pairs transfer at all.

Why not just make hard negatives by swapping words in real captions? Text-side swaps train the text encoder. Why is Winoground Hard? finds that the text side already separates the captions and that the failure lies where visual features meet language. We include a text-only hard-negative fine-tune at equal data size. If it transfers as well as our visual counterfactuals, the visual side of our pipeline is not needed.

Why not just close the modality gap with ModGap? ModGap raises MSCOCO clustering V-Measure from about 13.0 to 23.63, but nobody has measured it on minimal pairs. Its equilibrium argument concerns where the two modalities sit relative to each other, not whether "red" is attached to the right object. Running the fuzzer against a ModGap model answers the question directly. If ModGap's failure map shrinks to near nothing, closing the gap is the simpler fix, and we would say so.

Proposed Experiments

Discovery efficiency. For CLIP ViT-B/32 and SigLIP, each of the three generators (evolving, language model without coverage, fixed grammar) gets an identical budget of rendered pairs and a capped language-model spend. We plot distinct failing signatures against compute. A random sample of pairs from every generator is audited by humans for label validity. We expect the evolving generator to keep finding new signatures after the fixed grammar has saturated. If the curves overlap, the first claim is refuted.

Transfer. CLIP is fine-tuned on discovered hard pairs at ColorSwap's size of 1,400 and at 10× and 100× that size. We evaluate on Winoground and EqBen, with ColorSwap as a held-out skill. Zero-shot CIFAR-10 and MSCOCO retrieval are tracked to catch forgetting. For the synthetic-to-photo domain issue, a control fine-tunes on the same number of pairs from the fixed grammar. This separates the effect of broad coverage from the effect of simply seeing rendered data. The decisive result is a Winoground group-score gain that the ColorSwap-style narrow fine-tune and the fixed-grammar fine-tune do not produce. The absence of such a gain would support ColorSwap's view that the skills do not combine.

Gap versus composition. Across CLIP, SigLIP, ModGap and AlignCLIP-style models, we correlate each pair's modality gap with its margin between correct and swapped scores, within each failure signature. We also compare failure maps between models. This asks whether closing the gap, as ModGap and Mitigate the Gap advocate, or keeping it, as Mind the Gap suggests, changes compositional failures. Either outcome is informative. A null correlation would indicate that the gap and binding failures are separate problems.