When a vision-language model answers a question about a forty-page paper or a long stream of mixed images and text, we usually assume it found the relevant figure and read it. Nothing in current evaluation checks that assumption. Long multimodal benchmarks are built from public documents and images that may have been crawled into pre-training. One of them already shows that models answer about 57% of its questions without seeing the paper at all. So a correct answer at 50,000 tokens may reflect real long-range grounding, or it may reflect a prior the model brought with it. The familiar curves showing accuracy falling as context grows are confounded in the same way.
We propose a generator that builds evidence-swapped twins of long multimodal test items. Each twin keeps the document, layout, length, and question of an existing benchmark item, but replaces the specific visual or textual evidence so that the correct answer changes. A model that truly grounds its answer will follow the swap. A model leaning on memory or surface cues will repeat the original answer. The same engine also produces fully fresh documents from a hidden fact graph, with length, position, number of reasoning steps, and distractor density all under control. Together, the twins and fresh documents split long-context performance into three parts: what the model already knew, what it actually located in context, and how quickly that locating ability decays with length.
Core Claim
Static long-context multimodal benchmarks overstate how far into a context models can find visual evidence. They also misplace where the drop happens, because prior-based answering props up accuracy at long lengths. On evidence-swapped twins and fresh procedurally generated documents, we expect three things. First, the effective visual window, meaning the longest context at which accuracy stays within a fixed margin of short-context accuracy, will be shorter than static benchmarks suggest. Second, model rankings will change once accuracy is corrected for grounding. Third, the part of the decline usually blamed on attention dilution will turn out to be partly a contamination artifact that appears or disappears depending on how familiar the background material is.
The contribution is a measurement protocol whose central quantity is the grounding-dependence rate. This is the fraction of items answered correctly in their original form whose answer also correctly switches when the evidence is swapped. Its complement among swapped items is the prior-reliance rate: the fraction where the model gives the original answer even though the document now contradicts it.
Relation to Prior Work
Long-context grounding failures are well documented. Needle In A Multimodal Haystack shows that image needles degrade faster than text needles as interleaved contexts grow from 1k to 72k tokens. Visual Haystacks and Needle In A Video Haystack find lost-in-the-middle effects and a collapse on counting. PRISMM-Bench shows accuracy falling from snippet to whole-document settings. These papers motivate the proposal. All of them, however, use backgrounds that are static and potentially crawled, such as OBELICS-style text, MSRVTT and ActivityNet video, or public ICLR papers. PRISMM-Bench's own blind test, at roughly 57% accuracy with no paper shown, is the clearest evidence that this confound is real.
Dynamic generation supplies our method. DyVal generates reasoning problems from directed acyclic graphs, meaning graphs of facts where each conclusion depends on earlier ones. BeyondBench generates solver-verified algorithmic problems at very large scale. Minerva programmatically varies context length and distractor density, and separates search from stateful memory operations. All three are text-only. The multimodal dynamic benchmarks, PuzzleBench, NPHardEval4V, and KBE-DME, generate single images or evolve single-image visual question answering, so they have no long context. We reuse the graph-based generation of DyVal and the parametric control of Minerva, and move both into long interleaved documents.
The closest work is VideoNIAH, which also synthesizes needles inside long contexts. It differs from our design in two ways. It leaves the haystack static, and it inserts new content, which creates visible seams. We instead replace evidence that already sits in the document, and we treat background familiarity as an experimental variable rather than a fixed condition.
Contamination auditing constrains the design. Both Text and Images Leaked! (MM-DETECT) detects leakage through option-order sensitivity and slot guessing. It shows that leakage often comes from the base language model's text pre-training. This tells us that text-side priors, not just image memorization, must be controlled. MM-DETECT diagnoses short static benchmarks. It cannot say whether a particular correct long-context answer used the document. That item-level attribution is what the twins provide.
Research Program
The engine represents each item as a hidden fact graph. Leaf facts are bound to carriers: a photograph of an object class drawn from a held-out pool, a chart rendered from sampled values, or a sentence produced by a template grammar. Questions are compositions over the graph. They range from single lookup, to multi-step joins across carriers on different pages, to counting, to stateful tracking where later pages update earlier facts, in the spirit of Minerva.
To block the optical-character-recognition shortcut that Visual Haystacks warns about, images used as carriers contain no rendered text. Answers come directly from the graph, so ground truth is exact, as in DyVal.
The engine runs in two modes. In twin mode, it takes an existing item from MM-NIAH, PRISMM-Bench, or VNBench, finds the evidence that determines the answer, and swaps it. An image needle is replaced with a held-out photo of a different class. A plotted value is re-rendered so a different option becomes correct. A key sentence is edited. Everything else in the document stays the same. In fresh mode, the engine lays out an entire interleaved document from a new graph. Total length runs from 1k to 72k tokens to match MM-NIAH, and position, reasoning depth, and distractor similarity are swept.
A third, crossed condition places identical fresh needles into either crawled backgrounds or generated backgrounds. This isolates how familiar background material affects the decay curve.
For each model we report four quantities: original accuracy, grounding-dependence rate, prior-reliance rate, and the slope of fresh-mode accuracy against log context length, broken down by carrier modality and position. Each condition includes a blind setting with no document. On a sample of twins, two annotators check that the swap is the only change that matters and that the new answer is unique.
Baselines
The naive reference is static-benchmark accuracy as reported. On MM-NIAH, Gemini-1.5-Flash reports a 55.3% overall average and InternVL-1.5 reports 42.4% (Needle In A Multimodal Haystack). On PRISMM-Bench, GPT-5 with high reasoning reports 53.9% (PRISMM-Bench). On VNBench, Gemini 1.5 Pro reports 66.7% and LLaVA-OneVision-72B reports 58.7% (Needle In A Video Haystack). We rerun the open models on the original items under our harness, using the same prompts and the same decoding settings as for the twins, so that original and twin scores differ only in the evidence. The proposal beats this reference if grounding-corrected scores reorder at least one adjacent pair of models with bootstrap confidence intervals that do not overlap, or shift the estimated effective visual window by a bucket or more.
The strongest existing contamination tool is MM-DETECT (Both Text and Images Leaked!). No numeric Δ or Φ values are reported on Lacuna for it. We apply its option-order sensitivity test, item by item, to the same long-context items and the same models. This gives MM-DETECT's instance-level contamination flag as a competing predictor of which items are prior-answered. Our protocol wins if the prior-reliance flag from the twins separates items whose correctness survives the blind setting much better than MM-DETECT's flag does, measured by area under the curve against blind-correct labels.
The blind no-document baseline used by PRISMM-Bench is the simplest competitor. PRISMM-Bench reports about 57% blind accuracy with natural-language questions and about 34% with structured JSON questions. We run it on every item.
We also run a needle-insertion baseline in the style of VideoNIAH and MM-NIAH, matched for length and position. Here synthetic needles are inserted rather than swapped in, which tests whether seams change the measured curves.
Two ablations isolate the key ingredient. The first crosses identical needles with crawled versus generated backgrounds. The second uses real photographs as carriers versus rendered shapes, which separates contamination effects from synthetic distribution shift.
As a system-level reference, we also run a retrieval front end in which InternVL-1.5 is paired with a retriever, the setting where MM-NIAH reports 50.1%. This asks whether retrieval's gains also survive the swaps.
Why not just…?
Why not just run the blind baseline on existing benchmarks? The blind test bounds how much a benchmark can be answered from priors. It does not say whether a model that saw the document used it. A model can be correct both blind and with the document while ignoring the document entirely, or it can be correct with the document for real reasons on an item that happens to be blind-guessable. The twins resolve this item by item. This alternative is a real threat at the aggregate level: if blind accuracy alone predicts the corrected rankings, the twins add little. We report the rank correlation between blind-adjusted scores and grounding-dependence scores directly. A correlation near 1 would be a negative result for our method, and we would state it as one.
Why not just apply MM-DETECT-style perturbations to long benchmarks? Shuffling options and guessing masked words detect memorization of format and text. PuzzleBench shows that models fall to near chance under randomized options and collapse when answering directly. MM-DETECT itself traces leakage to text pre-training. Neither perturbation changes the evidence, so neither can show whether the answer follows the document. Our baseline comparison measures how often the two signals disagree.
Why not just insert synthetic needles into crawled haystacks? This is what MM-NIAH and VideoNIAH do, and the insertion is contamination-free for the needle itself. Two problems remain. VideoNIAH acknowledges that insertion leaves seams a model can detect from low-level artifacts. A familiar background may also change how attention is spread across the context. Our crossed background ablation tests the second problem directly. If crawled and generated backgrounds give the same decay slopes for identical needles, needle insertion is enough for measuring dilution. In that case we would recommend it and narrow our claim to item-level attribution.
Why not just let a frontier model write long multimodal tests? LongHalQA reports GPT-4o at 52.19% on long-context hallucination, with some models answering "Yes" up to 99% of the time (LongHalQA). A model-written answer key would bring exactly the errors we are trying to measure into the test set. KBE-DME also notes that generating with a large model is costly and introduces the generator's biases. Our answers come from a graph and are exact.
Proposed Experiments
Models include InternVL-1.5, LLaVA-OneVision-72B, Qwen2-VL-72B, and GLM-4.5V among open models, plus Gemini and GPT-4o-class systems where an API is available. The twin source sets are MM-NIAH retrieval, counting, and reasoning items; PRISMM-Bench document-level items; and VNBench-Long, where for video we swap needle clips with held-out clips. We build about 1,000 twins per source and 10,000 fresh documents.
Four checks decide the outcome. First, is the grounding-dependence rate on static long-context items well below 1, and does it fall with length faster than raw accuracy does? That pattern would show that priors hold up long-context scores. Second, is the effective visual window measured on fresh documents shorter than the one implied by MM-NIAH curves for the same models? Third, does the background-familiarity ablation change decay slopes when needles are identical? This separates contamination from dilution. Fourth, as a validity check: do model rankings from cheap fresh-mode documents match grounding-corrected twin rankings, with Spearman correlation of at least 0.8? If so, fresh mode can serve as a renewable benchmark that does not need a static source.
If swaps are followed nearly always and the background ablation shows no effect, we will conclude that contamination is not a meaningful confound for long multimodal grounding at current scales. That result would make existing dilution findings more trustworthy, and we would report it as such.
References
- Needle In A Multimodal Haystack
- Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs
- PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- Minerva: A Programmable Memory Test Benchmark for Language Models
- PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solving
- NPHardEval4V: A Dynamic Reasoning Benchmark of Multimodal Large Language Models
- Dynamic Multimodal Evaluation via Knowledge-Enhanced Benchmark Evolution
- Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMs
- LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models