A visual program is a short Python script that an LLM writes to answer a question about an image. It calls vision tools such as an object detector or a color classifier, and then combines their outputs with ordinary code. Recent systems improve these programs by running them, showing the intermediate values to a critic model, and letting a refiner rewrite the code. This proposal asks whether that repair loop can make things worse. The critic is itself a language model. When a correct program returns something surprising, such as "green" for a banana or "3" for a pair of shoes, the critic may read the surprise as a bug. The refiner may then rewrite the program until it returns what text statistics predict rather than what the image shows. If so, the loop that is supposed to fix errors instead trains the program to ignore its own perception. On old public benchmarks, where the expected answer and the expected program are both likely memorized, this failure would even look like improvement.
Our claim is that symbolic programs make this failure testable and preventable in a way end-to-end models do not. Every perception result in a visual program is an explicit return value. We can replace it with a counterfactual value (a different color, one more detection, a moved box), rerun the program, and check whether the answer follows. We propose to accept a repair only if the repaired program's answer depends on its perception calls at least as much as the original program's did. We will measure how often current repair loops pull answers toward language priors. We will test whether this cheap execution-level check stops the drift without giving up the real logic fixes. We will use scoring that holds up under contamination, so that memorized text cannot pass for improvement.
What already exists and what is missing
Three lines of work motivate the proposal. ViperGPT writes a visual program in a single pass by composing detection, depth and captioning models through Python APIs. It supplies our execution engine. Its "interventional explainability" disables whole perception modules to see how accuracy changes on average over a dataset. We instead intervene on individual return values inside one program on one image, and we use the result to decide whether to accept an edit. VDebugger trains a critic, which reads line-by-line variable traces to locate bugs, and a refiner, which rewrites the faulty line. It reports that about 58% of visual program failures are logic errors rather than perception errors, and gains of up to 3.2% on standard benchmarks. De-fine refines programs without training, using feedback from a VLM that inspects crops, an LLM that checks logic, and compiler messages. It raises GQA accuracy from 49.7% (ViperGPT) to 55.3%.
Neither refinement system asks whether its feedback is biased. Both evaluate only on static public sets (GQA, TallyQA, RefCOCO, OK-VQA). On such sets, an answer that matches the prior is usually also the correct one, so prior-driven repair would not show up.
The second line documents the bias itself and constrains our design. NaturalBench pairs two images with two questions so that the answer flips across images. It gives credit (Group Accuracy, G-Acc) only when all four cases are right, and shows that blind text-only models beat chance on ordinary benchmarks. Don't Just Assume; Look and Answer shows accuracy collapsing when answer distributions shift between train and test (VQA-CP). Evidence-RL supplies the core idea we borrow. It replaces visual tokens in an evidence region with their mean and rewards the model only when its confidence falls. It also reports that text-only feedback, such as an LLM acting as judge, produces reasoning that looks better while perception stays flat. Evidence-RL works on continuous tokens and changes model weights through reinforcement learning. We apply the same counterfactual logic to the discrete return values of a program at inference time, and use it as an acceptance test for code edits rather than as a training reward.
The third line, on contamination, determines how we evaluate. LiveCodeBench shows accuracy dropping sharply on problems released after a model's training cutoff. DyCodeEval shows that a deliberately contaminated 1B model rises from about 20% to about 80% on a static code benchmark but scores about 13% on rewritten versions of the same problems. Both are text-only and single-pass for code. Neither says anything about whether gains from refining visual programs are memorized. We do not build a new dynamic benchmark. Instead we use two cheaper controls, described below, that work directly on existing multimodal data.
Core Claim
We make two claims. First, execution-guided repair of visual programs causes measurable prior-induced drift. On images where the correct answer contradicts common sense, repair edits disproportionately change a correct answer into the answer a blind LLM would give. Second, a repair gate based on counterfactual dependency removes most of this drift while keeping most of the gains on standard benchmarks.
To make the gate precise, let $P$ be a program, $x$ an image, and $v_1,\dots,v_k$ the values returned by the perception calls when $P$ runs on $x$. For each $v_j$ we draw a type-matched counterfactual value $\tilde v_j$. A color is replaced with another color, a detection list gains or loses an element, a box is moved to a region not already covered by an existing box. We then rerun $P$ with $v_j$ forced to $\tilde v_j$. The evidence dependency of $P$ on $x$ is
$$E(P,x)=\frac{1}{k}\sum_{j=1}^{k}\mathbb{1}\left[P(x;\,v_j\leftarrow\tilde v_j)\neq P(x)\right],$$
which is the fraction of perception results whose change also changes the answer.
A proposed repair $P'$ is accepted if either of two conditions holds. It may fix a crash. Or it may satisfy $E(P',x)\ge E(P,x)$ without adding any hard-coded answer literal that the original program did not contain. Otherwise it is rejected, and the loop goes back to the last accepted program.
The gate needs no ground-truth label and no extra model calls, only extra executions of the program. The expensive vision tool outputs are cached, so each counterfactual rerun costs only the Python code. The gate does not judge whether a patch is correct; the critic still does that. It only blocks patches that make the program pay less attention to what it perceived.
Research Program
The work has three parts.
The first is measurement. We will instrument ViperGPT-style programs generated by open code models, then repaired by the VDebugger critic and refiner and, separately, by the De-fine feedback loop. For every accepted edit we record the answer before and after, $E$ before and after, and the answer of a blind LLM given only the question. From these we define the prior-pull rate: among edits that change the answer, the fraction that move a correct, image-supported answer to the blind answer. The comparison point is the same rate for edits that move a wrong answer to a correct one. Drift exists if the prior-pull rate is well above zero on prior-conflict data and clearly higher there than on GQA.
The second part is the gate itself. We will test variants that differ in how the counterfactual values are drawn: type-matched as above, uniformly random, or only on perception calls whose outputs the answer actually reads. This tells us whether the choice of intervention matters.
The third part is the contamination lens, applied through two controls. The first is paired group scoring on NaturalBench. Because each question appears with two images that require different answers, a program that has memorized the answer string cannot earn group credit. The second is a nulled-perception ablation. We rerun each pipeline with every vision tool returning an uninformative value: empty detections, or the most common label for that type. The accuracy that remains measures how much a pipeline's score comes from the code model's text knowledge rather than from the image.
We expect this remaining accuracy to be much higher on GQA than on NaturalBench. We also expect repair loops without the gate to increase it, meaning that repair converts perception-driven programs into prior-driven ones. This control is the multimodal-program analogue of the post-cutoff drops documented by LiveCodeBench. It needs no data newer than the training cutoff.
Repair budgets are capped at three rounds for every method. Yan et al. report that gains from execution feedback level off after two to four iterations, so further rounds mostly add chances for drift.
Baselines
The naive reference is single-pass ViperGPT with no repair. For GQA the file gives ViperGPT at 48.1% zero-shot in its own paper, and 49.7% as re-run inside De-fine. We will re-run it on the same tool stack for every benchmark rather than rely on those figures.
The strongest existing methods are De-fine and VDebugger. De-fine reports 55.3% on GQA, 75.2% IoU (overlap between predicted and true boxes) on RefCOCO and 57.1% on OK-VQA. VDebugger reports gains of up to 3.2% on standard benchmarks and 2.3% on COVR. They are the right competitors because they are the systems whose feedback we claim is biased. We will run both with their released critics and prompts, the same generator model, the same perception tools and cache, and the same three-round cap. They will be matched on program executions: because our gate adds cheap reruns, each baseline receives an equal number of additional sampled programs, chosen by its own critic.
No result for visual-program pipelines is reported in the file on NaturalBench or FREAK (a benchmark of questions whose images conflict with common-sense priors). For end-to-end VLMs, NaturalBench reports 20–40% G-Acc for leading models and about 30 points below humans for closed models. Evidence-RL reports a +14.22% gain on FREAK over its base VLM. We report these only as context, because the systems are different.
Beating the strongest baseline means two things together. On NaturalBench, the gated loop must reach higher G-Acc and a lower prior-pull rate than ungated VDebugger or De-fine. On GQA, TallyQA and RefCOCO, it must stay within one point of their accuracy. A gated loop that wins on NaturalBench but loses most of the GQA gain would count only as a partial result.
Two ablations isolate the key ingredient. The first is a gate with random rather than type-matched counterfactuals, which tests whether the design of the intervention matters. The second is an equal-compute rollback rule that reverts any edit whose answer matches the blind LLM's answer. This tests whether counterfactual execution does anything beyond simply penalizing agreement with the prior.
Why not just…?
Why not just give the critic the image and tell it to look carefully, or use a stronger VLM critic? This is the most natural fix, and we test it directly with GPT-4o-class critics given explicit grounding instructions. The existing evidence is against it. NaturalBench finds leading VLMs, GPT-4o included, at 20–40% G-Acc on exactly the prior-conflict cases where the critic would need to be right. Evidence-RL finds that text-level instructions and judge feedback leave perception unchanged. If a strong instructed critic nonetheless matches the gate's prior-pull rate at similar cost, the gate is unnecessary, and we will report that.
Why not just restrict repair to crashes and runtime errors, and never act on semantic critique? This is a real threat, because drift would come from semantic critique. It is our "crash-only" baseline. Its weakness is the one VDebugger identifies: most failures are logic errors that never crash, so crash-only repair should give up much of the GQA gain. If crash-only repair keeps most of De-fine's and VDebugger's GQA improvement while matching our NaturalBench results, the gate adds nothing, and we will narrow the claim to a diagnosis of the problem.
Why not just penalize answers that agree with a blind LLM? This is cheaper than counterfactual execution, but priors are often correct. Rejecting every prior-consistent edit should hurt GQA, where prior and truth usually agree. The blind-agreement rollback ablation measures this directly. The gate should do better because it examines how the answer is produced rather than what the answer is.
Why not just evaluate on VQA-CP-style shifted splits and skip the contamination controls? Shifted splits are static and public. DyCodeEval and LiveCodeBench show that static public sets end up measuring recall. Group scoring on paired images defeats answer recall by construction, and the nulled-perception ablation measures dependence on text knowledge directly. For the same reason we treat VQA-CP only as a secondary check.
Proposed Experiments and Decisive Checks
We evaluate on GQA, TallyQA and RefCOCO, the standard benchmarks where existing numbers are reported, and on NaturalBench and FREAK as the prior-conflict sets. The generator is an open code model of the kind VDebugger trains. Perception tools are fixed and cached across all conditions. Metrics are accuracy or IoU, NaturalBench G-Acc, prior-pull rate, mean $E$ of the final programs, and accuracy under nulled perception.
Three results decide the proposal. First, drift must be real. The ungated repair loops should show a prior-pull rate on NaturalBench that is clearly higher than on GQA, and their NaturalBench G-Acc should fall at or below single-pass ViperGPT even as their GQA accuracy rises. If ungated repair helps NaturalBench as much as GQA, the premise fails and the project stops at that negative result. Second, the gate must separate the two effects. It should lower the prior-pull rate and raise G-Acc while staying within a point of ungated accuracy on GQA, and it should beat both the crash-only and the blind-agreement baselines at equal compute. Third, the contamination lens should show that ungated repair increases accuracy under nulled perception on GQA. That would mean some of the published repair gains come from text knowledge rather than from looking at the image, while gated repair leaves that accuracy unchanged.
References
- ViperGPT: Visual Inference via Python Execution for Reasoning
- VDebugger: Harnessing Execution Feedback for Debugging Visual Programs
- De-fine: Decomposing and Refining Visual Programs with Auto-Feedback
- NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
- Evidence-RL: Towards Evidence-intensive Visual Reasoning
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation