Measuring Memorization Through the Text–Image Gap in Multimedia Event Extraction

When a news article comes with a photo, the photo often shows things the article never says: which vehicle carried the troops, what weapon was used, who stood at the podium. Earlier work on the MultiMedia Event Extraction benchmark (M2E2) found that about a third of news images contain such "visual-only" event participants. We propose to use that gap as a built-in test for memorization. A model that names a visual-only participant correctly while the image is withheld cannot have read it from the input. It either guessed from general world knowledge or remembered the source story, for example because Voice of America articles from 2020 and their photos were in its training crawl. By comparing withheld-image recall on M2E2 against a matched set of post-cutoff news, and by swapping in images from other events, we can separate real cross-modal reading from memory. We can also measure how much of a model's reported multimedia score is inflated.

This matters now because the strongest M2E2 results come from large vision-language models that were almost certainly trained on web news. These results are scored either with strict string matching or, increasingly, with language-model judges that accept synonyms and broader terms. The judge's leniency adds a second problem. A judge that accepts "vehicle" for a gold "armored truck" will reward generic guesses made without the image, so a model can look grounded when it only has good priors. The proposal therefore treats the semantic judge as part of what is being measured, not as a trusted instrument.

Core Claim

The text–image asymmetry defines a set of answers that the text alone cannot support. That lets us build a contamination probe for open-ended structured extraction without generative image editing. The quantity we care about is how often a model recovers visual-only arguments with the image removed or replaced, compared with how often it does so on matched news it could not have seen. For a model $m$, let $R_{\text{full}}$ be its recall of visual-only arguments with text and the original image, $R_{\text{blind}}$ its recall with the image removed, and $R^{\text{fresh}}_{\text{blind}}$ the same blind recall on post-cutoff articles, which measures the prior-guessing floor. We define the memory-attributable share of visual-only recall as

$$ M = \frac{R_{\text{blind}} - R^{\text{fresh}}_{\text{blind}}}{R_{\text{full}}}, $$

so $M$ near zero means visual-only recall is either grounded or explainable by generic priors, and large $M$ means reported cross-modal gains partly reflect remembered stories. The image-swap condition checks $M$ independently. We replace the original photo with a photo from another event of the same type whose visual-only argument differs, then count how often the model still outputs the original article's argument. We call this the persistence rate $P$. A grounded model should follow the new image. A memorizing model should keep the old answer.

Positioning Against Prior Work

Three papers motivate the question by establishing strong results on M2E2 without any contamination audit. WASE aligns text meaning graphs with image situation graphs. It reports gains over text-only systems and the observation that 33% of news images hold arguments never mentioned in the text. MMUTF recasts argument extraction as template filling, reaching 38.2% textual argument F1. Benchmarking and Improving LVLMs on Event Extraction from Multimedia Documents shows that Qwen2.5-VL models reach 71.8% F1 on few-shot visual event detection and 65.0% multimedia argument F1 after low-rank fine-tuning. We take their benchmark and the one-third visual-only statistic as our starting point. None of them asks whether a 2020 VOA corpus is already inside these models.

The contamination work supplies the method and also limits it. Both Text and Images Leaked! (MM-DETECT) shows that text parts of multimodal benchmarks leak during text pretraining and let models answer visual questions without looking. We borrow the idea that memorized answers break under meaning-preserving changes. However, its probes are option shuffling and masked-word guessing, and neither applies to open event-argument outputs. Vision-Language Bootstrapping (VLB) edits images with inpainting and edits questions to reduce overlap with pretraining sets. It reports image overlap up to 84.46% and MME contamination falling from about 33% to about 10%. It relies on an unaudited model judge and costly diffusion edits. Our main probe needs no generation: blind and swapped conditions are recombinations of real data, and inpainting appears only as a secondary check. DyVal shows that static leaders can collapse on freshly generated instances. That motivates our post-cutoff control set, although we do not generate synthetic tasks.

The judge literature limits how we score. VALOR-EVAL shows that two-stage language-model matching detects hallucinations far better than keyword metrics (82.35% vs. 5.88% for CHAIR). It also deliberately accepts broader terms, which is exactly the leniency that could reward blind guessing. HaloQuest reports r = 0.99 agreement between its Gemini judge and humans, but only on VQA, and it never checks for bias toward its own model family. Preference Leakage finds judges favoring related models by up to 37.1% on Arena-Hard. The Silent Judge finds verdicts shifting up to 30% on irrelevant cues, with rationales that never mention them. Both results are text-only. We use them to justify blinding the judge to condition and model identity and to cross two judge families, not as a primary research target.

Research Program

The first component is the probe set. For every M2E2 argument we label whether it is text-only, image-only, or in both. For image-only arguments we check with two annotators that the text genuinely does not entail them. We then build three input conditions per article. The first has the full text and image. The second removes the image, either by giving text only or by substituting a blank frame, and we report both variants because some models behave oddly without an image. The third swaps in an M2E2 or post-cutoff image of the same event type whose visual-only argument differs; we verify that the swapped image stays plausible with the text and that its argument differs from the original. Because M2E2 has only 245 articles, the pool of visual-only arguments will be modest. We will therefore annotate a fresh set of roughly 150 to 200 multimedia news articles published after the evaluated models' training cutoffs, using the same ontology. This set serves two purposes: it estimates the prior-guessing floor, and it provides a direct uncontaminated score to validate $M$.

The second component is the judge protocol. Models produce open-ended role fillers. A judge compares each filler only against the gold argument text, never against the image, the article, or the condition label. Its rubric asks at what specificity the match holds: exact, synonym, or broader term. We report recall at "specific" and "any" leniency separately. This lets us show directly how broader-term acceptance inflates blind recall. We run two judges from different model families plus strict string F1. On a human-labeled subset of about 1,500 judgments we measure agreement per condition, because a judge that works well on full-input outputs may behave differently on blind guesses. We also test whether judge verdicts differ by evaluated-model family at matched human labels, extending Preference Leakage's question to this setting.

The third component is attribution. For each evaluated model we compute $M$ and $P$ on M2E2 and compare the model's M2E2 score with its fresh-set score. As a descriptive covariate, we also run MM-DETECT-style masked-word guessing on the VOA article text to test whether article-level memorization predicts $M$.

Baselines

The naive reference point is the standard reported number: full-input F1 on M2E2 with strict matching, taken at face value as cross-modal skill. The best reported multimedia argument result in the file is 65.0% F1 for LoRA-fine-tuned Qwen2.5-VL-7B, and the best visual event detection result is 71.8% F1 for few-shot Qwen2.5-VL-72B, both from Benchmarking and Improving LVLMs on Event Extraction from Multimedia Documents. The file reports no number for recall of visual-only arguments specifically (no reported result found on Lacuna for visual-only argument recall on M2E2). We re-run this model under that paper's prompting and fine-tuning settings so that our full-input numbers are comparable before we add conditions. Our probe "beats" this reference if it identifies a gap between M2E2 and fresh-set performance that full-input F1 cannot reveal. Concretely, the rank order of models by $M$ should predict how much each model's score drops from M2E2 to the fresh set, with rank correlation clearly above zero, while full-input M2E2 F1 predicts the fresh-set score worse.

The strongest existing contamination detector is MM-DETECT's masked-word guessing on perturbed text from Both Text and Images Leaked!. We apply it to M2E2 article text, using the same back-translation step and the same models, and score it the same way. It is the natural competitor because it already captures unimodal text leakage, which is the mechanism we suspect. Our probe wins if $M$ predicts the M2E2-to-fresh drop better than MM-DETECT's accuracy drop. It also wins if it detects inflation in models where MM-DETECT is silent, for instance models that remember the event from other coverage rather than the VOA text itself. If MM-DETECT's score predicts the drop equally well, we will report that our probe adds only interpretability, not detection power.

The ablation that isolates the key ingredient removes the prior-guessing correction and uses $R_{\text{blind}}/R_{\text{full}}$ alone. Without the fresh-set floor, strong-prior models should be flagged falsely, and we expect this version to correlate worse with the fresh-set drop. A second ablation replaces the specificity-aware judge with a lenient broader-term judge in the style of VALOR-EVAL. We expect that judge to raise blind recall and compress the differences between models. A VLB-style inpainting variant, applied to a small subset of images to change the visual-only object, serves as a costlier alternative to image swapping for estimating $P$. We run it under the same per-item compute budget accounting so that cost is compared honestly.

Why not just…?

Why not just run MM-DETECT's existing perturbations on M2E2? It is our main competitor and we run it. Its probes answer a different question, though: whether the article text is memorized. They do not answer whether that memory inflates extraction of image-only arguments. A model can memorize an article without recalling the photo's contents, or recall the event's vehicles from other coverage without memorizing that article's wording. The comparison of how well each measure predicts the M2E2-to-fresh drop settles which signal matters. If MM-DETECT wins, the proposal collapses into "apply MM-DETECT," and we will say so.

Why not just annotate a fresh post-cutoff benchmark and stop using M2E2? This is a real threat. For a single evaluation round, a fresh set is the cleanest answer, which is why we build one. But it has costs. It goes stale at the next training crawl, as MM-DETECT observes for newer benchmarks. It gives no diagnosis of which models relied on memory, and it is too small to be the only test. The blind and swap probes can be rerun cheaply on any static corpus. Our fresh set exists mainly to validate them. If fresh-set scores alone explain everything and $M$ adds no predictive value, the probe is not worth adopting.

Why not just score with strict string F1 and avoid judges entirely? Strict matching removes judge bias, but it undercounts paraphrased fillers. VALOR-EVAL's gap between CHAIR and semantic matching shows how badly keyword metrics fail on open outputs. Undercounting also varies with a model's verbosity, which would contaminate cross-model comparison. We report strict F1 alongside both judges. If $M$ and $P$ rank models the same way under strict F1, the judge becomes optional, and we would count that as a useful result.

Why not just use VLB-style inpainting to create counterfactual images? Editing a vehicle or weapon inside a news photo is exactly the fine-grained, safety-sensitive content that Benchmarking and Improving LVLMs on Event Extraction from Multimedia Documents reports models refusing or mislocalizing. VLB itself flags the inference cost. Swapping real images avoids both edit artifacts and cost. We still run inpainting on a subset to test whether the persistence rate $P$ agrees across the two methods.

Proposed Experiments

We evaluate the Qwen2-VL, Qwen2.5-VL, and DeepSeek-VL2 families used in the LVLM benchmarking paper at several sizes, plus MMUTF as a small non-generative reference. Each is run few-shot and, for the 7B models, with LoRA fine-tuning on M2E2 training data, under full, blind, and swapped conditions on both M2E2 and the fresh set. Metrics are recall and F1 by argument type (text-only, image-only, both), at specific and lenient judge settings and under strict matching, together with $M$, $P$, and judge–human agreement per condition.

Three decisive checks follow. First, if models exceed the fresh-set blind floor on M2E2 by a margin larger than bootstrap confidence intervals over articles, and $P$ is well above zero, then reported M2E2 visual-argument gains are partly memory. The size of $M$ tells how much. Second, the rank correlation between $M$ and each model's M2E2-to-fresh drop, compared against MM-DETECT's correlation, tells us whether the gap-based probe is a better detector or only an explanation. Third, if blind recall rises substantially when the judge switches from specific to lenient matching while human-labeled blind recall does not, that is direct evidence that broader-term semantic judging turns prior guessing into apparent grounding. Future multimedia judges would then need specificity-aware rubrics. A null result is also informative: $M$ near zero across models, $P$ near zero, and no gap between M2E2 and fresh scores would be the first evidence that M2E2 remains a valid test of cross-modal reading.