The Picture Is Fresh, the Knowledge Base Is Late

Real-world facts change before any knowledge base records them. A stadium is renamed, a company redesigns its logo, a shop changes owners. Wikipedia can trail the news by about a year for new entities (How much is Wikipedia Lagging Behind News?). During that window, a photograph is often the only up-to-date evidence anyone has: the new name is already painted on the wall. A vision-language model (a language model with an image encoder attached, which we call an LVLM) should read the wall. We suspect it usually does not. Its language backbone has seen the old name thousands of times in pretraining, and LVLMs are known to give text far more weight than image content.

This note proposes to measure that failure directly, explain it, and test which fix works. The core idea in two sentences: build questions from real, dated changes to entities whose new state can be read off an image, then check whether the model reports what it sees or what it memorized. Then test whether the error is a knowledge problem, which retrieval or editing would fix, or a modality-weighting problem, which methods that boost attention to the image would fix.

Core Claim

We claim that answers which state outdated facts about recently changed entities are, in a measurable share of cases, a form of modality bias rather than missing knowledge. The model can read the new fact from the image when the old fact is absent from its memory. It fails when a strongly memorized old fact competes with what it reads. Three predictions follow, and they make the claim falsifiable.

First, the rate of stale answers should rise with how strongly the old fact is memorized. We will measure memorization strength with a text-only fill-in-the-blank probe. Test of Time found this kind of probe to be the format that remains sensitive to memorized training text: it reports a 77.5% influence-function hit rate for cloze questions, which falls to 17.5% after paraphrasing.

Second, methods that rebalance attention toward the image should reduce stale answers.

Third, adding retrieved text from a knowledge base snapshot taken before the change should make things worse. A text-dominant model will trust stale retrieved text over a fresh image.

The third prediction ties the knowledge-base lag problem to modality bias in a way no paper in our file tests. Retrieval is the standard cure for outdated knowledge, but during the lag window it can be the source of the error.

Research Program

The closest work is MMKU-Bench. It studies how to update a multimodal model's knowledge across text and images. It reports "knowledge inertia": updated facts are harder to learn than facts that were never known, even when the correct context is supplied. It also reports large gaps when an update is made in one modality and tested in the other.

Our study differs from MMKU-Bench in three ways. First, MMKU-Bench supplies the update as training data or as golden text context. In our setting, the update arrives only as visual evidence at inference time, and nothing is trained. Second, its updated facts are counterfactuals generated with GPT-4o. Ours are real, dated changes. Third, it diagnoses the gap but does not intervene on how attention is split between text and image.

SearchLVLMs adds live web search to LVLMs and reports outperforming GPT-4o by about 30% accuracy on its UDK-VQA benchmark. It assumes the retrieved text is fresh, which is precisely what fails during the lag window.

MMBench-Live continuously harvests fresh images to avoid benchmark contamination. Its authors note that models still show implicit memorization of common concepts, and its scores do not separate a reading failure from memory overriding perception. We borrow its key mechanism: a "vision-blind" verifier that accepts a question only if tools such as OCR can derive the answer from the image.

The program has three parts.

Benchmark construction. We will build a set of change pairs from knowledge-base revision histories, such as Wikipedia and Wikidata edit logs of the kind aligned with news by Fetahu et al.. Each pair is an entity whose visible, text-readable attribute changed: a renamed venue, a rebranded storefront, a changed sign or logo wordmark. We restrict to attributes readable by OCR so the verifier can confirm that the new answer is in the image.

Each change yields four items:

  • a dated image showing the new state, harvested from Flickr or image search with post-change timestamps, as in MMBench-Live;
  • a matched image of the old state;
  • a text-only prompt with no image, which measures the model's prior;
  • a matched control entity whose same attribute did not change.

The old-state image and the text-only prompt separate "cannot read" from "will not believe." Each item also records three dates: the change date, the date the knowledge base recorded it, and the model's training cutoff. This makes lag a measured variable rather than a nuisance.

Contamination works in our favor in this design. The old fact is supposed to be memorized, and its memorization strength is the covariate we measure. The new answer post-dates training, so the new answer cannot have leaked.

To test whether synthetic edits behave like real ones, we will add a counterfactual subset in the style of MMKU-Bench. In these items an image editor paints a fake new name onto an old-state photo.

Analysis. For each model we will regress the stale-answer rate on three quantities:

  • memorization strength, measured as the cloze log-probability of the old fact;
  • entity popularity, using page views, which Liang et al. found predictive of update behaviour;
  • the layer-wise modality attention ratio of LLaVA Steering, a measure of how much attention goes to image versus text tokens.

The analysis also has a mechanistic component. DMAS found that "truthfulness" heads in LLaVA-1.5 sit in a different layer from visual-perception heads. We will test whether, on stale answers, the truthfulness directions point toward the old fact. If they do, steering toward "truth" with vectors fitted offline would entrench stale answers.

A minimal intervention. Existing rebalancing methods apply a fixed strength. The file notes that this strength is hyperparameter-sensitive, and that too much intervention degrades fluency. We will test a gated variant that scales visual rebalancing by the measured strength of the competing prior. The gate costs one extra text-only forward pass per question, which contrastive decoding already computes. If the gate adds nothing over fixed-strength rebalancing, we will report that as a finding.

Baselines

The naive reference is the unmodified model answering with the image. We will run LLaVA-1.5-7B, Qwen2.5-VL-7B, and InternVL2.5-8B, the backbones used by RBD, VASparse, DMAS, and MMKU-Bench, with greedy decoding and a fixed prompt. Its stale-answer rate on the conflict items is the quantity everything else must reduce.

Re-Balancing Contrastive Decoding (RBD) is the strongest existing competitor on the modality-bias question. It subtracts logits produced from a noise-corrupted image, which isolates text-prior bias, and it amplifies strongly attended visual tokens. On LLaVA-1.5-7B it reports 88.0% POPE accuracy and reduces sentence-level CHAIR from 20.8% to 17.8%.

We will also include VASparse, which reports instance-level CHAIR falling from 7.22% to 5.82% on LLaVA-1.5, and MoReS, which reports 88.2% POPE. DMAS reports +94.66 MME points on LLaVA-1.5.

None of these methods has a reported result on stale-fact conflict. For our benchmark there is no reported result found on Lacuna, so we will run all of them ourselves. Each method gets the same images and the same backbone. Its hyperparameters will be tuned with an equal budget of 20 configurations on a held-out development split of change pairs. We will report each method's published default alongside its tuned setting.

Fresh-text retrieval in the style of SearchLVLMs is the strongest competitor on the knowledge question. We will run it under three retrieval conditions:

  • a knowledge-base snapshot dated after the change;
  • a snapshot dated before the change, which mimics the lag window;
  • no snapshot.

All three use the same filtered-passage budget.

Instruction prompting tells the model to "answer from what is visible in the image."

The gating ablation compares the gated intervention with the same rebalancing method at its best fixed strength.

The intervention beats the baselines only if three conditions hold together. It must lower the stale-answer rate on conflict items by more than the variation across random seeds and prompts. It must not raise the error rate on unchanged control entities. And it must keep POPE and MME within one point of the unmodified model.

Why not just…?

Why not just retrieve fresh information, as SearchLVLMs does? During the lag window, no fresh text exists to retrieve. The knowledge base is behind by weeks to more than a year (Frank et al.; Fetahu et al.). Retrieval with a post-change snapshot is a genuine threat on items outside the window, and the three-condition retrieval baseline measures that directly. Our claim survives only if pre-change retrieval increases stale answers relative to no retrieval.

Why not just tell the model to trust the image? This is cheap and could work. If prompting matches rebalancing on conflict items without raising errors on controls, then the error is a matter of instruction-following rather than attention, and we would drop the intervention claim. We expect it to fall short for two reasons. Hallucination snowballing shows models deferring to their own text over images they can see (Investigating and Mitigating the Multimodal Hallucination Snowballing). And the prompt itself is more text in a model that already over-weights text.

Why not just use existing contrastive decoding? This may well be sufficient, and it is the most likely outcome to deflate the method component. If RBD or VASparse at a tuned fixed strength matches the gated variant, the contribution narrows to the benchmark and the diagnosis. Those two parts still stand: no existing method has been evaluated on real stale-fact conflicts, and the retrieval-harm prediction does not depend on the intervention.

Why not just fine-tune or edit the new facts in? MMKU-Bench reports that full fine-tuning on updated knowledge drops Qwen2.5-VL's MME score by over 89%. It also reports that the editing methods GRACE and WISE do not scale to continuous batches of updates. More fundamentally, editing requires knowing about the change, and during the lag window nobody has recorded it yet.

Proposed Experiments

We will build an initial set of change pairs drawn from changes after each model's cutoff, plus an equal number of unchanged controls. Every item will pass the vision-blind OCR verifier, and a sample will be audited by humans, against the 96.06% correctness that MMBench-Live reports.

We will report three primary metrics: the stale-answer rate on new-state images, accuracy on old-state images (which confirms the model can read), and the text-only prior answer. General capability will be monitored on POPE and MME.

Three checks decide the outcome.

Check 1: memorization drives stale answers. The stale-answer rate must increase with the cloze memorization strength of the old fact while old-state reading accuracy stays flat. If the stale-answer rate does not track memorization, the modality-bias explanation fails, and the errors are better explained as plain reading failures.

Check 2: retrieval hurts during the lag window. Pre-change knowledge-base retrieval must raise the stale-answer rate above the no-retrieval condition, while post-change retrieval lowers it. This is the result that connects knowledge-base lag to text dominance.

Check 3: rebalancing helps more as priors get stronger. Attention rebalancing must cut stale answers mainly on high-memorization items, and must do so without overriding correct answers on controls. We will also test whether DMAS truthfulness vectors push toward old facts, by comparing stale-answer rates with and without its truthfulness component.

Comparing the real changes against the synthetic counterfactual subset will show whether GPT-4o-style counterfactual benchmarks under- or over-estimate knowledge inertia. That answers a limitation MMKU-Bench itself acknowledges.