Does Memorization Fill the Middle? Measuring Multimodal Lost-in-the-Middle on Post-Cutoff Science

Long-context models are known to lose track of information placed in the middle of a long input. Accuracy is high near the start and the end and sags in between, which is often called "lost in the middle." Multimodal models show the same sag when they search through many images. Almost every measurement of this sag, however, has been taken on old public data such as MS-COCO from 2014 or the OBELICS web corpus, which these models have very likely seen during training. If a model already half-remembers a document, it can answer a question about the middle of that document without actually reading it. The measured curve then mixes two abilities: reading a long context and recalling training data.

This proposal separates those two abilities using time. We will build long, interleaved figure-and-text contexts from scientific papers and pose questions that require reasoning across a figure and a claim in the text. We will then run the same items on models whose training cutoffs fall at different dates, so a fixed item is "unseeable" for some models and "possibly seen" for others. This turns contamination from a nuisance into a controlled variable. It lets us ask whether memorization artificially fills in the middle of the U-curve, and whether published fixes for positional bias still help once memorization cannot.

Core Claim

Our hypothesis is that, for multimodal reasoning over real scientific content, the middle-position penalty is larger on material that post-dates a model's training cutoff than on material that pre-dates it. The gap between the two should be widest in the middle positions, because recall from training data rescues exactly the positions where attention to the context is weakest. Static long-context benchmarks would then understate how badly models read the middle. Mitigations evaluated on static data, such as retrieval or contrastive decoding, may also overstate their gains.

The design that makes this testable is a difference-in-differences across models. Items are fixed; what varies is whether each item falls after a given model's cutoff. We fit

$$\text{logit}\,P(\text{correct}_{i,m,d}) = \alpha_i + \gamma_m + \beta_1\,\text{post}_{i,m} + \beta_2\,\text{mid}_d + \beta_3\,\text{post}_{i,m}\cdot\text{mid}_d,$$

where $\alpha_i$ is a per-item difficulty term and $\gamma_m$ a per-model skill term. $\text{post}_{i,m}$ indicates that item $i$'s source paper appeared after model $m$'s cutoff, and $\text{mid}_d$ indicates that the evidence sits at a middle depth $d$. The per-item terms absorb any topic or difficulty difference between old and new papers, because the same item serves as "pre" for one model and "post" for another. The claim lives in $\beta_3$. A significantly negative $\beta_3$ means memorization fills the middle. A $\beta_3$ near zero means positional curves on static data are trustworthy, which is a useful negative result in its own right.

Relation to Prior Work

Three benchmarks established that multimodal models lose the middle, and all three rest on static data. Needle In A Multimodal Haystack hides synthetic needles in OBELICS documents up to 72k tokens and finds a middle dip plus a "vision gap" for image needles. Visual Haystacks searches for COCO images among up to 10,000 and sees sharp dips at middle positions. MMNeedle stitches COCO sub-images into grids and finds an edge bias. These papers motivate our question but do not answer it. Their needles are retrieval targets rather than reasoning steps, and their haystacks are old enough to be in training data.

Our evaluation form borrows from Ref-Long, which shows that text models fail to attribute facts to the right document in long inputs. We extend that kind of referencing to figures.

Two papers bring temporal freshness to multimodal evaluation. MMBench-Live harvests fresh images cheaply but stays single-image. VeriSciQA builds verified questions from recent paper figures, again single-figure. We reuse VeriSciQA's approach to building questions: extract claims, then run a check that the question cannot be answered without the figure. We differ from both papers by placing that content at controlled depths in long contexts and by using cutoff variation across models as the experimental lever.

Two papers measure contamination in multimodal models with local perturbations on single images. Both Text and Images Leaked! shuffles answer options and masks words in paraphrased captions, and finds that text-only leakage lets models answer visual questions. Vision-Language Bootstrapping edits images and inserts distracting text. Neither asks how contamination interacts with position, and we treat both as competing ways to remove contamination.

The temporal-firewall logic itself comes from LiveCodeBench and LiveBench, both text-only. Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? supplies the motivating evidence that scores drop on post-cutoff material.

Two papers constrain the mitigation arm. Positional Contrastive Decoding recovers middle tokens in text models. It does so by subtracting the predictions of a deliberately near-sighted copy of the model, created by over-rotating its low-frequency position encodings. The Mirage of Performance Gains shows that multimodal contrastive decoding gains largely come from a "Yes" bias exploiting label imbalance, and vanish on generative tasks. That result warns us to report decoding gains only on balanced, open-answer items.

Research Program

The data source is arXiv papers with figures, harvested month by month over a window that straddles the published cutoffs of a panel of open and closed multimodal models. We will restrict each month to matched fields and figure types so that the per-item terms are not asked to absorb wild shifts.

For each target paper we generate two question types with a VeriSciQA-style pipeline. Single-figure readouts serve as a perception control. Cross-reference questions require joining a claim stated in the text with a quantity read from a figure located elsewhere, for example "which curve in the figure corresponds to the ablation the authors call harmful." Every item passes two checks. A vision-blind check confirms it cannot be answered from text alone. A closed-book check confirms that a no-context query to the model panel fails. We keep the closed-book check's results rather than discarding them, because they define one of our baselines.

Contexts are built by concatenating the target paper with other papers from the same month, so that distractors are equally fresh. Total lengths range over roughly 8k, 16k, 32k, and 64k tokens. The target paper's position is moved across five depths. For cross-reference items, we additionally vary the separation between the claim and the figure, so that one or both pieces of evidence land in the middle.

To separate the vision gap from the reasoning gap, a text-description arm replaces each figure with a written description of its content. This follows the logic of GeoSQA, which found that reasoning failed even when diagrams were described in words. Model cutoffs are taken from model documentation, padded by a buffer of a few months. As a robustness check, we also fit the cutoff as an unknown changepoint per model.

Baselines

The naive reference point is the standard practice: measure the positional curve on static, pre-cutoff content only and treat it as the model's reading ability. Concretely, this is our pre-cutoff split under the same contexts, depths, and prompts. The proposal beats this reference if the post-cutoff middle penalty differs from it with a confidence interval on $\beta_3$ that excludes zero.

The strongest existing way to remove contamination without time is a closed-book plus vision-blind filter on static items, the check used by VeriSciQA and MMBench-Live. It is the strongest competitor because it is cheap and directly removes items whose answers are recalled. We run it on the identical pre-cutoff pool, discard every item any panel model answers without context, and recompute the curves. Temporal isolation is worth its cost only if the filtered pre-cutoff curves still show a smaller middle penalty than the post-cutoff curves. If they match, the filter suffices, and we will say so.

The perturbation competitor applies option shuffling and masked-caption paraphrasing from Both Text and Images Leaked! and distractor insertion from Vision-Language Bootstrapping to the pre-cutoff items, with the same item counts.

For reference levels from prior benchmarks, MM-NIAH reports 55.3% overall for Gemini-1.5-Flash, and 42.4% for InternVL-1.5, rising to 50.1% with retrieval. Ref-Long reports GPT-4o at 14% exact match on a 24K-token task against 92% for human annotators. We will rerun available models on MM-NIAH to anchor our scale against these figures.

For the mitigation arm, the strongest method is positional contrastive decoding. Its paper reports raising Llama-3-8B on RULER at 16k from 64.40 to 69.04 F1, and InfiniteBench 8k accuracy from 72.0% to 79.0%. We will apply its over-rotation to the language backbone's one-dimensional positions in open models, alongside a retrieval baseline, with identical prompts and a fixed small grid for its strength parameter tuned on a held-out month. A gain counts only if it holds on post-cutoff, open-answer items.

The key ablation is the text-description arm. It isolates whether any contamination-by-position interaction is specific to visual evidence or also appears when figures are replaced by words.

Why not just…?

Why not just filter static items with a closed-book check? The filter catches items whose answers are recalled outright. It does not catch partial familiarity: a model that vaguely knows a paper's story can locate and interpret a middle figure with far less reading. This alternative is a real threat, and it is built in as a baseline. If filtered static curves match post-cutoff curves, the proposal's main message becomes "filtering suffices for positional studies," and we will report it that way.

Why not just perturb existing benchmarks? The bootstrapping paper reports that dynamic perturbation cut measured contamination on MME from about 33% to about 10%, which is a reduction but not an elimination. The perturbations are also local edits to single images, whereas the leakage we care about concerns a whole document's content. The perturbation arm will show directly whether perturbed static items reproduce the post-cutoff middle penalty.

Why not just use synthetic needles, which are uncontaminated by construction? Synthetic needles solve contamination for retrieval, and MM-NIAH already uses them. Reasoning over real scientific figures, however, cannot be faked with an inserted cartoon. MM-NIAH itself notes that its answers depend only on the needle, not on reasoning across the context. Our cross-reference items exist precisely to fill that gap.

Why not just fix the middle with retrieval or contrastive decoding and skip measurement? MM-NIAH found retrieval largely ineffective for image needles. The Mirage paper shows that multimodal contrastive decoding gains can be artifacts of label imbalance. Whether either fix survives on uncontaminated reasoning is one of our measured outcomes, not an assumption.

Proposed Experiments

The decisive experiment is the fit of the model above across the full panel, items, depths, and lengths, with $\beta_3$ and its bootstrap interval as the primary result. This is reported separately for single-figure and cross-reference items and for the figure and text-description arms. We expect $\beta_3$ to be most negative for cross-reference items with visual evidence at middle depth.

The second check repeats the fit after closed-book filtering and after perturbation, to see which cheap method, if any, recovers the post-cutoff curve.

The third check reports the gains from retrieval and positional contrastive decoding separately on pre- and post-cutoff items. It tests whether gains measured on possibly-seen data overstate gains on unseeable data. We will also break out answer distributions to rule out the bias artifact documented in the Mirage paper.

Finally, a validity check confirms that per-item difficulty is stable. A model whose cutoff post-dates every item should show no "post" effect at all, and any nonzero $\beta_1$ for such models signals a construction confound that must be fixed before the main claim is read.