Multi-Object Interventions on Fresh Scenes to Separate Seeing from Guessing

Vision-language models (systems that attach an image encoder to a large pretrained text model) often say an object is in a picture because the surrounding scene makes it likely, not because they see it. A knife and a plate make the model report a fork. The benchmarks used to measure this have two flaws. First, their headline numbers can be moved by shifting the model's overall tendency to answer "yes," with no change in what it perceives. Second, they are built from public images such as MSCOCO, which have leaked into training data. This proposal builds an evaluation on freshly composed scenes. Whether the target object is present and which context objects surround it are set at random, jointly and independently. A simple decomposition of the model's "yes" log-odds then separates three things: a global answer bias, the real effect of the pixels, and the pull of each context object together with how those pulls combine.

The core claim is that grounding should be measured as the causal effect of visual evidence on the answer, estimated under multi-object interventions, rather than as accuracy under single-object swaps. Three predictions follow. Bias-shifting fixes such as contrastive decoding and "answer yes if possible" prompts will change the intercept but not the visual-evidence coefficient. Context pull will be superadditive, meaning several co-occurring cues together pull harder than the sum of their separate pulls. Single-swap protocols cannot detect that by construction. And the measured strength of the language prior will be larger on familiar, possibly memorized COCO scenes than on fresh ones.

Where This Sits

The motivating evidence comes from three places. Causal-HalBench models hallucination with a structural causal model that has a co-occurrence confounder. It edits MSCOCO images with FLUX-controlnet and SAM, replacing exactly one object per image, for example a person with a bear. It reports, for instance, a counterfactual-object hallucination rate of 6.8% for LLaVA-NEXT-8B. It shows that context confounds perception, but it stays inside the atomic intervention constraint (only one variable changes at a time), it uses static COCO images, and it inherits inpainting artifacts. The Object Multiplicity Study reports that hallucination rises when several objects are involved at once, and Concept Association Bias describes models treating images as bags of co-occurring concepts. Both point to interaction effects that a single swap cannot reveal.

The second motivating paper is a constraint. The Mirage of Performance Gains shows that the gains of Visual Contrastive Decoding on the balanced POPE benchmark come mostly from a shift toward "yes" plus a truncation step that turns sampling into greedy search. A prompt saying "Answer Yes if possible" reaches 90.2% accuracy on LLaVA-13B (COCO Random), against 89.2% for VCD. We take this as a design requirement: any grounding metric we report must be invariant to a global bias shift. Our decomposition is built to meet that requirement, and it also gives a way to settle the VCD-versus-Mirage dispute directly rather than by benchmark choice.

The third motivating paper is Both Text and Images Leaked!. It introduces MM-DETECT and finds that contamination of multimodal benchmarks comes substantially from the text-only pretraining of the language backbone, with older benchmarks leaking more. DyVal and CrossWordBench show the value of procedurally generated test items. DyVal works on text-only reasoning graphs and CrossWordBench on puzzle grids, and neither involves causal interventions on scene content. We borrow procedural freshness from them and apply it to hallucination scenes.

From causal representation learning we borrow a design principle, not a theorem. Identifying Linearly-Mixed Causal Representations from Multi-Node Interventions shows that environments in which several variables are clamped at once can identify latent factors with about $2\lceil \log d \rceil$ environments for $d$ variables, if the environments form a separating system (every pair of variables is clamped differently in at least one environment). General Identifiability and Achievability for Causal Representation Learning shows that observational baselines remove the need for faithfulness assumptions. Our setting is easier than theirs, because we observe the scene variables directly. We do not claim to identify hidden variables inside the model. What carries over is that multi-variable hard interventions, arranged as a separating system and anchored by an observational baseline, extract more causal information per environment than one-at-a-time changes, and they also expose interactions.

Core Measurement

Each test item is a composed scene $s$ paired with the question "Is there a $T$ in the image?" for a target category $T$. The scene variables are as follows. $v \in \{0,1\}$ records whether $T$ is actually present. $c_1,\dots,c_d \in \{0,1\}$ record whether each of $d$ context objects is present, where the context objects are those that co-occur with $T$ most strongly in COCO statistics, matching POPE's adversarial selection. $b$ is a background scene type, which is either congruent or incongruent with $T$. All of these are set by intervention: the generator places them, and nothing in the scene is left to natural co-occurrence. We fit

$$\operatorname{logit} P(\text{yes} \mid s) = \beta_0 + \beta_V v + \sum_{j} \gamma_j c_j + \sum_{j<k} \delta_{jk} c_j c_k + v \sum_j \eta_j c_j ,$$

The terms are interpreted as follows. $\beta_0$ is the global answer bias. $\beta_V$ is grounding, meaning the effect of the pixels of $T$ when no context is present. $\gamma_j$ is the pull of context object $j$. $\delta_{jk}$ measures superadditive joint pull. $\eta_j$ measures how much visual evidence is discounted or amplified when context agrees with it. Prior dominance means the context terms are large relative to $\beta_V$, or $\eta_j$ is negative so that pixels matter less when the context already predicts the answer. A pure bias shift changes only $\beta_0$, which is why this decomposition is immune to the Mirage critique. For open-weight models, $P(\text{yes})$ comes from the "Yes" and "No" token logits. For API models, it is estimated from repeated sampling.

Research Program

The first component is a scene composer that avoids generative inpainting. Objects are segmented photographic cutouts drawn from a privately held pool of newly photographed and licensed images that have never been posted publicly. They are composited onto held-out backgrounds with randomized position, scale, lighting harmonization, and occlusion. Each object instance and background is used in only a small number of scenes, and the pool is refreshed for each release. This rules out instance-level memorization. Category-level priors stay intact, which is intended, because those priors are what we want to measure. We state plainly that the categories and question templates are not novel. The protection against contamination covers images and layouts, not concepts.

The second component is the intervention design. An observational environment draws $v$ and the $c_j$ from their COCO co-occurrence frequencies. Intervention environments clamp sets of variables according to a separating system over $\{v, c_1,\dots,c_d\}$, and the remaining variables are randomized independently. Pooled together, these environments decorrelate every pair of variables and support estimation of all main effects and pairwise interactions. A matched-budget atomic design, which changes one variable at a time from the observational baseline in the style of Causal-HalBench, serves as the ablation.

The third component is a text-only arm. The same question is posed with a short text description of the context objects and no image, which estimates the backbone's pure language prior for each configuration. Comparing $\gamma_j$ from images with the text-only pull shows how much of the context effect is linguistic and how much is visual association.

The fourth component is a layer-wise readout. For open models, the same regression is fit to "yes"/"no" logits obtained by projecting intermediate hidden states through the output head. This tests the reported deep-layer suppression of visual evidence (Layer Degradation Probing; the attention-ratio analysis in LLaVA Steering) in terms of causal coefficients rather than attention weights. This readout is secondary. If intermediate projections turn out to be uninformative, we drop it.

The fifth component is a contamination arm. The identical design is run on scenes built from COCO validation backgrounds and object crops. Those images are suspected to be leaked, but they are composited by the same pipeline. We then compare the coefficients with those from fresh scenes. For each backbone, we also compute the MM-DETECT drop $\Delta$ on COCO captions. Across models, we ask whether a larger COCO-minus-fresh gap in context pull goes with stronger leakage. With roughly eight to ten models, this test is correlational and has little statistical power, and we will report it that way.

Baselines

The naive reference point is the unmodified model under greedy decoding, scored by POPE-style accuracy, F1, and yes-ratio on the same fresh scenes. It shows what conventional metrics report on our data. The second reference is the same model scored by AUROC of $P(\text{yes})$ on COCO POPE Adversarial. AUROC is already invariant to bias, so it is the strongest existing alternative metric. On POPE itself the reported numbers are accuracy and F1. MiniGPT-4 drops from 78.86 to 71.37 F1 between the Random and Adversarial splits (Evaluating Object Hallucination in Large Vision-Language Models). No reported AUROC result was found on Lacuna for POPE.

The methods compared are VCD (Leng et al.), which reports up to +7.4 F1 on POPE MSCOCO Random; the Prompt-Based Adjustment from The Mirage of Performance Gains, which reports 90.2% accuracy against VCD's 89.2% on LLaVA-13B COCO Random and is the strongest training-free competitor on existing benchmarks; and MoReS (LLaVA Steering), which reports 88.2 on POPE and is the strongest trained method in the file that claims to restore visual attention. VCD and PBA run on LLaVA-1.5-7B/13B, the models on which they were originally reported, with the authors' hyperparameters. VCD is additionally run with its truncation step switched off, to separate the two mechanisms Mirage identified. MoReS is trained with its published recipe on LLaVA-665k. We give a LoRA baseline the same data and the same number of GPU-hours, with a small matched hyperparameter sweep (equal number of configurations per method) tuned on a held-out set of fresh scenes disjoint from the test pool.

What counts as beating these baselines differs by component. Our claim about the metric is supported if PBA moves $\beta_0$ while leaving $\beta_V$ unchanged within its confidence interval, while POPE accuracy rates PBA as an improvement. A mitigation method counts as genuinely improving grounding only if it raises $\beta_V$, or brings $\eta_j$ toward zero, beyond seed-level variation. The key ablation is the atomic design at matched image budget. Ground truth for that comparison comes from a dense full-factorial reference set of about 10x the budget, built for two target categories. The multi-object design should recover $\beta_V$ and $\gamma_j$ with lower error and should detect any $\delta_{jk}$ that the atomic design cannot estimate.

Why not just…?

Why not just extend Causal-HalBench to swap two or three objects? That would be a multi-node intervention, but it would keep inpainting artifacts and static COCO images, and it would still measure accuracy, which bias shifts can move. The design change alone gains little without bias-invariant coefficients and fresh scenes. The atomic-versus-multi-object ablation at matched budget tests the design part of the claim directly. There is a real risk here: if $\delta_{jk}$ and $\eta_j$ come out near zero for every model, then single swaps already recover the main effects, and the multi-object design is only an efficiency gain. We would report that outcome as a negative result about interactions.

Why not just report AUROC on POPE Adversarial? AUROC removes the "yes" bias, which addresses Mirage. In natural images, however, a present target co-occurs with its context, so a high AUROC can come from reading the context rather than the object. Only interventions that decorrelate $v$ from the $c_j$ can tell these apart. To check this, we compute AUROC on our observational environment and compare it with $\beta_V$ from the intervention environments. A large gap between them is the evidence that AUROC conflates the two. If the two agree closely, AUROC is enough and our metric adds little beyond the interaction terms.

Why not just run MM-DETECT to handle contamination? MM-DETECT (Song et al.) estimates whether text leaked, but it does not say whether leakage changes how much a model trusts the pixels. The paired COCO-composited and fresh arms measure that effect directly. MM-DETECT's $\Delta$ is used as the per-model covariate in that analysis, so it complements our measurement rather than replacing it.

Why not just use CLEVR-style rendered scenes? Rendering creates a domain gap that could lower $\beta_V$ for reasons unrelated to language priors. Photographic cutouts reduce the gap. We also include a control in which each target appears alone on a neutral background, which should give near-ceiling $\beta_V$. If it does not, the composer is the problem and the coefficients are not interpreted.

Decisive Checks

We evaluate LLaVA-1.5, LLaVA-NEXT-8B, InstructBLIP, Qwen-VL, Kimi-VL-A3B, GPT-4o, and Gemini 1.5 Pro on about 40 target categories, each with $d=6$ context objects, roughly 300 fresh scenes per category, and a matched COCO-composited arm. Four results would decide the proposal. The first is metric validity. PBA must leave $\beta_V$ unchanged. If it raises $\beta_V$, the decomposition is leaking bias, and we stop and repair it before going further. The second is a reading of contrastive decoding. If VCD without truncation leaves $\beta_V$ unchanged, Mirage's account holds in causal terms. If $\beta_V$ rises, VCD does more than Mirage credits, and that would be a substantive correction to Mirage. The third is interactions: whether a meaningful fraction of target categories show $\delta_{jk}>0$ or $\eta_j<0$ significantly after multiple-comparison correction. The fourth is the contamination effect: whether context pull $\gamma$ is systematically larger on COCO-composited scenes than on fresh ones within the same model. If $\gamma$ is not larger, then contamination does not drive prior dominance for this task, which would also be worth reporting.