Vision-language models such as CLIP often ignore word order. They give nearly the same score to "the horse is eating the grass" and "the grass is eating the horse," so they behave like a bag of words. An attractive workaround is visual programming: a language model writes a short Python program that calls a detector, a depth estimator and a question-answering model, then checks relations explicitly with code. Nothing guarantees the generated program is any better, though. A program can pass the whole caption to a CLIP-like module, or compare the wrong two boxes, and so reproduce the same order-blindness. Iterative code refinement, where a model rewrites code in response to execution feedback, has a clear weakness here. A crash produces a traceback, but a program that runs and gives the wrong answer produces no error, and at test time there is no label to say it is wrong.
We propose a label-free test for exactly this failure, borrowed from software engineering's metamorphic testing. Metamorphic testing checks how a program's outputs should relate across transformed inputs when no correct answer is known. Each caption is paired with automatically generated variants. Role swaps ("grass eating horse") must flip the program's verdict on the same image. Paraphrases and swaps of symmetric relations ("next to") must leave it unchanged. When a program violates either rule, the system feeds the violation back to the code model as a failing unit test. The feedback includes intermediate values: which boxes were found, which module produced which score, and whether the two programs differed only in perception or also in logic. The model then repairs the program. The core claim is that this swap-test signal is what makes refinement loops work for compositional matching. If so, closed-loop visual programs should beat both open-loop programs and text-only decomposition on adversarially cleaned compositionality benchmarks.
What exists and what is missing
Two systems established that code can compose vision modules. ViperGPT generates Python against an API of GLIP, BLIP-2 and MiDaS. VisProg does the same with a library of more than twenty neural and symbolic modules. Both generate programs in a single pass. VisProg attributes about 16% of its GQA errors to incorrect program logic and leaves automated self-correction as future work. Neither was tested on minimal-pair benchmarks built to expose bag-of-words behaviour. These papers supply our program format and module library. We add the closed loop and the stress-test evaluation.
Refinement loops in software motivate the closed loop and also warn against it. Raw tracebacks fed back to the model repair pipelines in CAAFE. Multi-agent review in MAGIS raises GPT-4's SWE-bench resolution from 1.74% to 13.94%. Rollback to the last stable state in MEIC counters the "new error trap," where edits introduce fresh bugs. Iteration gains can still plateau after a few rounds (Yan et al.). All of these loops use failure signals that come from a compiler, a test suite or a simulator. Our domain has no such oracle, so the real contribution is constructing one from the structure of the caption.
The closest approach from the bag-of-words side is CECE. It has a language model write entailed and contradicted statements, then scores each with a frozen vision-language model. It reports a +19.2% Winoground group score over the prior best, and +12.9% on EqBen. CECE is text-only and runs in one pass, so a hallucinated entailment or an inconsistent score goes unchecked. Our swap tests are related in spirit, since both use caption variants. The difference is that we use variants as consistency checks on an executable program and repair the program when the checks fail, rather than averaging scores over the variants.
The diagnostic literature constrains the design. CLIP's single-modality embeddings encode attribute binding almost perfectly, up to 99% under linear probes, and the loss happens in the cross-modal match (Koishigarina et al.; Diwan et al.). This argues for programs that query localized regions rather than scoring whole captions. SugarCrepe showed that rule-based negatives let text-only models win, and that NegCLIP's gains vanish on cleaned data. Any method that generates its own perturbations must therefore be evaluated on benchmarks it did not help construct.
Research Program
The system has three parts.
The first is the program writer. An open code model writes a verification function for a caption, which returns a score that the caption holds in the image. It uses a ViperGPT-style API with box localization, attribute and relation queries on crops, and depth. Geometric relations are written as explicit comparisons of box coordinates and depth values.
The second is the swap-test generator. A language model rewrites the caption into role swaps, attribute swaps and paraphrases. It labels each variant with the verdict relation it expects relative to the original: flip, or stay the same. Including variants that must stay the same is essential. Without them, the loop could "pass" simply by making the program erratically sensitive to word order.
The third is the repair loop. The program and its variants are executed on the image, violations are reported along with their intermediate values, and the model rewrites the program. Rollback is borrowed from MEIC: a revision is kept only if it reduces total violations without breaking previously satisfied tests. The loop stops after a fixed number of rounds or once no violations remain.
Before building the loop, we run a diagnostic study that is useful on its own. For one-shot ViperGPT programs on Winoground and SugarCrepe, we measure what fraction are order-invariant, meaning they give the same verdict to a caption and its role swap. We then split these failures into two kinds. Logic invariance means the program delegates the whole phrase to a single module. Perception invariance means the detector returns the same box for both entities. If logic invariance is rare, the loop has little to fix and the project's premise weakens. That makes this measurement the first gate.
Baselines
The naive reference points are zero-shot CLIP and one-shot ViperGPT-style programs using the same modules and code model. The original Winoground paper reports that most models fall below the 16.67% chance level on group score, for example FLAVA at 14.25%.
The strongest training-free competitor is CECE. The file gives only its relative gain, not an absolute Winoground score; no reported result found on Lacuna for CECE's absolute group score. We will therefore rerun CECE ourselves with the same language model and the same scoring vision-language model that our modules use. Our method has to beat that rerun.
The strongest overall reference is a large generative vision-language model. GPT-4V reports 90.95% overall and 83.13% on the SWAP subsets of SugarCrepe (Hsieh et al.). This is a ceiling for comparison rather than a like-for-like competitor.
The training-based reference points are NegCLIP, reported at 81% on ARO VG-Relation (Yüksekgönül et al.), and the LABCLIP linear adapter, reported at 95% on CLEVR (Koishigarina et al.). We use public checkpoints or retrain them under the reported recipes.
Three ablations isolate the key ingredient. The first is a traceback-only loop, where repair is triggered only by crashes and empty detections. The second is swap-rejection sampling: draw N programs at matched cost and keep the one with the fewest swap-test violations, with no rewriting. The third is the full loop without tests that expect the verdict to stay the same.
All programmatic methods share the same code model, modules, prompt-tuning budget and total model calls per image-caption pair, and report calls and wall-clock time. Beating the baselines means two things. On SugarCrepe relation and SWAP subsets, Winoground group score and PUG:SPARE spatial items, the full loop must exceed both CECE and compute-matched swap-rejection sampling by more than seed-to-seed variance. It must also not lose accuracy on GQA.
Why not just…?
Why not just use CECE's text decomposition? CECE is strong and much cheaper, so it is a real threat. Its weaknesses are on paper rather than measured: it cannot check hallucinated entailments, and spatial relations come down to a vision-language model reading a sentence, which is exactly where bag-of-words failures persist (Diwan et al.). The matched rerun, broken down by subset, settles it. If CECE matches us on spatial and relation subsets as well as attribute subsets, programmatic repair adds nothing and we will say so.
Why not just sample several programs and pick the most consistent one? This is the most dangerous alternative, because it uses our test without our loop. Swap-rejection sampling at matched compute is therefore a primary ablation. If it ties the full loop, the contribution shrinks to "swap tests are a good program selector." That is still useful, but it would mean the iterative refinement direction contributes nothing. The prediction that separates the two is that rewriting should help most on perception-invariant failures, where every sample shares the same detector error and only a rewrite can route around it.
Why not just train a linear adapter or fine-tune on hard negatives? LABCLIP is cheap and solves attribute binding on CLEVR, but its authors report that it does not solve spatial relations and costs ImageNet accuracy. NegCLIP's gains disappear on SugarCrepe. We include both, and expect them to compete on attribute subsets but trail on spatial ones. If the adapter matches us on PUG:SPARE spatial items, the geometric-code argument fails.
Why not just ask GPT-4V? On SugarCrepe it may simply be better, and we do not claim otherwise. The claim is narrower: small open modules plus a closed loop can recover most of that gap, with an inspectable reason for each verdict. Winoground's visually difficult items, where CLIP succeeds 0% of the time, will show whether perception rather than composition is the limiting factor for everyone.
Decisive checks
The evaluation covers Winoground (text, image and group scores), SugarCrepe broken down by subset, ARO VG-Relation and VG-Attribution, EqBen, and CLEVR and PUG:SPARE for spatial relations. GQA serves as a guard against regression on general question answering.
Four results decide the project. First, the diagnostic must show that a substantial share of one-shot programs are order-invariant through their logic, not only their perception. Second, the full loop must beat swap-rejection sampling at matched compute. Third, removing the tests that expect unchanged verdicts must produce more variance-driven "passes" with no accuracy gain, which confirms the hacking guard is needed. Fourth, gains per round must be tracked, to show whether swap-test feedback delays the plateau reported for compiler feedback.
To rule out leakage, swap variants on Winoground are generated by the language model rather than taken from the paired caption, and we report results both ways. SugarCrepe's own negatives are never used as tests.
References
- ViperGPT: Visual Inference via Python Execution for Reasoning
- Visual Programming: Compositional visual reasoning without training
- Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
- MEIC: Re-thinking RTL Debug Automation using LLMs
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Natural Language Inference Improves Compositionality in Vision-Language Models
- CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
- Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality
- SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
- When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?