When the Diagram Lies: Measuring Whether Vision-Language Models Trust the Image or the Supplied Causal Graph

A common recipe for improving visual causal reasoning is to hand a vision-language model (a model that reads images and text and answers in text) an explicit causal graph. A causal graph is a diagram of arrows saying which variables in a scene drive which others. The model then answers questions such as "if the light moves, what else changes?" Several benchmarks report that supplying the correct graph helps. In practice, though, any graph handed to a model was written by a person or extracted by another model, and it will sometimes be wrong. Causal inference has a long record showing that downstream answers inherit graph errors, and that leaving out a true arrow tends to do more damage than adding a spurious one. Nobody has asked the multimodal version of this question. When the supplied graph contradicts what the image plainly shows, does the model follow the image, the graph, or neither?

We propose to answer that question with controlled graph corruptions on synthetic scenes where the true causal structure is known exactly. Because a wrong graph and the true graph imply different answers to the same question, each answer the model gives can be attributed to one of three sources: the visual evidence, the supplied text graph, or the model's pre-training prior. This turns graph misspecification into a probe of the multimodal reasoning gap. If models reliably side with a wrong text graph over contradicting pixels, then their "visual causal reasoning" is text-structure following, and gold-graph benchmarks overstate it. We then test one targeted remedy: fine-tuning on corrupted graphs whose correct labels follow the image. We test it against the obvious cheaper fixes.

Core Claim

Our hypothesis is that current vision-language models defer to a supplied causal graph even when before-and-after intervention images contradict it. We further hypothesize that their sensitivity profile differs from the one causal estimation theory predicts. Theory says omitted edges should be the most damaging and extra edges comparatively benign. We expect models instead to copy whatever arrows are present, so extra and reversed edges will hurt about as much as omissions. We also expect the damage to concentrate on variables with several descendants, since CausalVLBench already finds models fail there even with no graph. The new contribution is the measurement: a per-item decomposition of answers into image-following, graph-following and prior-following, made possible by controlled corruptions of known graphs. No existing paper reports this decomposition.

Where This Sits

Three papers motivate the question by showing that supplying a gold graph changes behaviour, but none perturbs that graph. ViLCaR injects hand-verified causal graphs for natural VQA scenes into Qwen2.5-VL-7B's prompt. This raises causal attribution and inference scores while answer accuracy stays flat (0.763 to 0.768). CausalVLBench compares only two conditions, true graph versus no graph, on Pendulum, Water Flow and Causal Circuit. CELLO prompts the model to extract a graph before answering, and with this step GPT-4o goes from 0.59 to 0.70. Each of these papers assumes the graph is right. We relax exactly that assumption.

Two papers constrain the design. CAUSAL3D shows that models succeed on familiar physics but fail on "hypothetical" scenes that use the same graph shapes with arbitrary rules. It also shows they ignore visual evidence that contradicts their priors. We therefore must use hypothetical scenes; otherwise "following the image" cannot be separated from "reciting textbook physics." The Abstraction Gap finds that fine-tuning on 45,000 gold chain-annotated examples does not close the gap between fluent text and correct causal chains. This warns that simply adding more correct-graph supervision is unlikely to help. It also motivates our choice of labels that conflict with the graph rather than agree with it.

BridgeVLM supplies the strongest method. It predicts a graph from images internally and restricts attention to that graph's edges. It explicitly leaves open what happens when the induced graph is wrong. Our corruption protocol applies directly to its routing mask if its code is available.

On the causal side, two papers provide the theory our measurements are compared against. Returning the Favour shows that a structural prior drops below an unconstrained baseline when the graph is wrong. Neural Bayesian Network Understudy ties the gains from independence constraints to the correctness of the graph. Together they supply the reference prediction, including the omission-versus-commission asymmetry, that we test vision-language models against.

Research Program

The core item is an intervention question with an answer that can be computed from the graph alone. The model sees a before image and an after image of a scene in which one variable was manipulated. It also receives a text graph and the identity of the manipulated variable $X$. It is asked which other variables changed.

Under any graph $G$, the implied answer is the set of descendants of $X$, written $D_G(X)$. The true graph $G^*$ gives the visually correct set $D_{G^*}(X)$, which the after image actually shows. A corrupted graph $G'$ gives a possibly different set $D_{G'}(X)$.

We focus on items where the two sets differ. For those, let $R$ be the set of variables the model reports. We define graph deference as

$$\delta = \frac{|R \cap (D_{G'} \,\triangle\, D_{G^*}) \cap D_{G'}| + |(D_{G'} \,\triangle\, D_{G^*}) \cap D_{G^*} \setminus R|}{|D_{G'} \,\triangle\, D_{G^*}|},$$

where $\triangle$ is symmetric difference, the set of variables on which the two graphs disagree. In words, $\delta$ is the fraction of disagreeing variables on which the model sided with the wrong graph. A value of $\delta = 0$ means it followed the image everywhere, and $\delta = 1$ means it followed the graph everywhere. A prior-following answer is identified by running the same item on the matched familiar-physics scene with no graph supplied.

Corruptions are generated by four operators: delete one true edge, add one spurious edge, reverse one edge, and delete a common parent so that a confounded pair looks causally linked. The operators are applied at one, two and three edits. We stratify by how many descendants the manipulated variable has and by graph size, two to five variables.

We serialize each graph in three formats: arrow strings, adjacency lists and JSON. This rules out the possibility that deference is an artifact of one notation, a gap The Abstraction Gap left open.

The second component tests whether the behaviour can be changed. We fine-tune Qwen2.5-VL-7B on corruption-conflict data drawn from CAUSAL3D's training scenes. In this data, roughly half the supplied graphs are corrupted, and the target answer is always the visually correct set, with an edge-level rationale ("the shadow did not change, so the arrow light→shadow is not supported here"). The point is that the label sometimes contradicts the graph. The model can only score well by checking arrows against pixels.

Baselines

The naive reference point is the same model given the image pair and no graph. This measures what visual evidence alone yields. A second reference is the gold graph condition, which reproduces the setup of CausalVLBench and ViLCaR. Their gold-graph gain is the quantity a wrong graph could erase or reverse. If a corrupted graph drops accuracy below the no-graph condition, the graph is acting as misinformation in the sense of Returning the Favour.

The strongest existing method is BridgeVLM. On CausalVLBench it reports 54.4% intervention accuracy against 33.2% for prompt-level supervision. On Causal3D it reports 92.3% counterfactual accuracy against 81.0%, both from From Prompts to Tokens. It is the strongest competitor because it does not trust a text graph at all; it induces its own graph from images. That makes it the natural candidate for being robust to misleading supplied graphs.

If its weights are available, we will run it on our items in two ways. First, with its own induced graph. Second, with its routing mask overwritten by our corrupted graphs, which measures whether hard-wired routing is more or less brittle than prompt injection. If the weights are not available, we compare only to its reported numbers on the original splits and say so. No reported result was found on Lacuna for BridgeVLM, or any model, under supplied-graph misspecification.

The prompting baseline is CELLO-CoT, which raises GPT-4o from 0.59 to 0.70 on CELLO. We run it with the corrupted graph placed where its graph-identification step would go.

The key ablation for our fine-tuning has two arms. One trains on identical corrupted graphs with labels that follow the graph rather than the image. The other trains on the same number of gold-graph examples. All three arms share data volume, steps, learning-rate sweep and base model.

Beating the baselines means two things. Our fine-tuned model must reach lower $\delta$ than every baseline on held-out hypothetical scene families. It must also lose no more than a point or two of gold-graph accuracy relative to gold-graph fine-tuning.

Why not just…?

Why not just perturb the graph and report accuracy drops? Accuracy alone cannot tell whether a model failed because it followed the wrong graph, ignored both inputs, or fell back on physics priors. These failures call for different fixes. The deference decomposition, together with the hypothetical-versus-physical scene pairing, is what separates them, and it costs nothing extra to compute.

Why not just tell the model the graph might be wrong and ask it to check? This is a real threat, and we test it directly as a "skeptical prompt" condition and a self-critique condition. Existing evidence is mixed. CAUSAL3D reports that models override visual evidence that conflicts with priors. MATH-Vision reports that chain-of-thought sometimes lowers multimodal accuracy, and both findings suggest asking for verification may not help. If the skeptical prompt matches fine-tuning on $\delta$, we report that models can verify graphs but do not do so by default. That is still the central diagnostic finding, and we would drop the fine-tuning claim.

Why not just skip the supplied graph and let the model infer one, as BridgeVLM does? Induced graphs are themselves misspecified. BridgeVLM's directed-edge F1 of 75.1% means roughly a quarter of its edges are wrong, and CausalVLBench shows that prompted models infer graphs poorly from interleaved image pairs. Overwriting BridgeVLM's mask with controlled corruptions measures whether internal routing inherits the same brittleness. If it does not, that is a strong result in BridgeVLM's favour, and we will report it as such.

Proposed Experiments

The primary testbed is CAUSAL3D's eleven hypothetical scene families, with two to five variables and known structural equations. We use them both to render intervention pairs and to compute $D_{G^*}$ exactly. Training and test scene families are disjoint. CausalVLBench's Pendulum, Water Flow and Causal Circuit serve as an out-of-distribution test set with physical semantics. Evaluated models are Qwen2.5-VL-7B, GPT-4o and Gemini. Each condition uses at least 2,000 disagreement items, with bootstrap confidence intervals on $\delta$.

As a transfer check on natural images, we corrupt ViLCaR's causal graphs by deleting the attribute node the answer depends on, or by inserting a distractor relation. We then ask whether ViLCaR's flat answer accuracy under gold graphs becomes a drop under corrupted ones. A drop would show that the graph is being used even though gold graphs did not move accuracy.

Three results would be decisive. First, $\delta$ well above 0.5 for prompted models on hypothetical scenes would establish graph deference. Second, comparable $\delta$ for added and reversed edges versus deleted edges would show the asymmetry predicted by causal estimation theory does not carry over. Third, a fine-tuned model with low $\delta$ on unseen scene families, where the graph-following-label ablation shows high $\delta$, would show that conflict-labelled supervision rather than data volume drives the change. If none of the prompted models defers, the hypothesis is wrong, and the paper becomes a positive robustness result for visual grounding.