Reading Stale Charts with Fresh Facts

A chart is a snapshot. A bar chart of quarterly revenue, a map of case counts, or a line plot of an interest rate is correct as of the day it was drawn. Later, some of its numbers get revised, extended, or replaced. A multimodal model is a model that takes an image and text together. When such a model is asked a question that needs the current state of the world, it has to combine three sources: what the chart shows, what a retrieved document says has changed since, and what the model itself memorized during training.

We propose to study and fix one specific failure in this setting. The model must decide, value by value, which numbers in the image an external update supersedes. It must also notice which numbers are untouched, and then compute over the merged result. Prior work has measured parts of this failure separately. Multimodal models ignore retrieved evidence that contradicts their memory. They answer chart questions from text shortcuts instead of reading the plot. They fail at multi-step arithmetic unless it is made explicit. No one has measured what happens when all three pressures meet, and that combination is how obsolescence shows up in real document reasoning.

Core Claim

Our hypothesis is that, when a chart is stale and an update is supplied, the dominant error is neither misreading the chart nor bad arithmetic. It is mis-scoping the update. Sometimes the model ignores an update that applies, which we call stale answers. Sometimes it applies an update where it doesn't belong, or answers from the update text without consulting the chart at all, which we call grounding-free answers. We further claim that making the merge explicit fixes most of this.

The fix works as follows. The model produces a small intermediate "ledger": a table read from the chart in which each cell keeps its visual anchor (series colour, axis position, legend label). Each update sentence is then linked to specific cells, or explicitly to none. A short program computes the answer over the merged table. The new ingredient is the scoping step, not the derendering. That makes the claim testable: an ablation that keeps everything except scoping should fall back toward the stale-answer and grounding-free failure rates of plain prompting.

What Exists and Where This Sits

The motivating evidence comes from four papers.

Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models (MMKC-Bench) tests nine multimodal models on retrieval conflicts. It finds that they cling to internal memory more than text-only models do, and more so as they scale. They detect conflicts about 75–79% of the time yet still fail to use the evidence. However, its conflicts are counterfactual edits to atomic encyclopedic facts, it proposes no remedy, and it involves no multi-step reasoning over the image.

Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details shows that chart accuracy reaches as high as 79% with informative titles and drops to the 30% range when titles are removed or misleading. This tells us retrieved update text is likely to become a new shortcut rather than a constraint.

Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset attributes 42.2% of GPT-4V failures to reasoning errors and 31.9% to vision recognition errors. It also reports that chain-of-thought prompting sometimes hurts multimodal models.

The closest method papers each cover one piece and leave out the piece we need.

Reliable Lifelong Multimodal Editing: Conflict-Aware Retrieval Meets Multi-Level Guidance (CARML) keeps a frozen model and retrieves edits from an external store. It reaches 99.3% reliability after 1,000 sequential edits on LLaVA-V1.5, against under 1% for weight-editing MEND. This is why we do not consider weight editing. But CARML's hard logit correction requires knowing the target token in advance, and its tasks are single-fact visual question answering and captioning. A computed answer over a partly updated chart has no pre-known target token.

REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignment trains models to locate the atomic "pivots" a multi-hop answer depends on, and contrasts against patch-shuffled images. Its pivot idea is the closest relative of our scoping step. But it targets encyclopedic entity questions, and patch shuffling destroys exactly the coordinate geometry a chart's meaning lives in.

Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs and Synthesize Step-by-Step supply our method components. The first contributes chart-to-table derendering as a training task and distilled rationales. The second contributes executable program rationales. Neither considers external updates.

BeaconChart: Verifier-Guided Corrective Instruction for Chart Understanding adds a planner, a perception agent and an independent verifier, and supplies the idea of re-checking perceived values. It too works on self-contained charts.

On the obsolescence side, Mitigating Temporal Misalignment by Discarding Outdated Facts predicts how long a fact stays true in order to decide when to retrieve. It is text-only and treats the decision as whole-answer confidence rather than per-value scoping. We borrow its framing of comparing an "as of" date against elapsed time when an update carries dates.

Research Program

The StaleChart benchmark. We first build a benchmark from ChartQA (ChartQA) and ChartX, using the source data tables that underlie the charts. For each chart we write updates as short dated statements. Some revise a cell ("the 2021 figure for France was revised to 4.2"). Some add a new period that is not drawn. Some are distractors that look relevant but do not apply: a different entity, an older date than the chart's, or a different metric. We then generate questions in three classes. Some questions need an updated value combined with untouched chart values, such as a difference, an average, or a ranking. Some depend only on untouched values. Some depend only on values targeted by distractors.

Because we hold both the old and the new table, every question has three exactly computable answers. The first is the correct merged answer. The second is the stale answer, computed from the original chart only. The third is the update-only answer, computed from whatever the update text alone permits. The stale and update-only answers give direct numeric analogues of MMKC-Bench's original-answer and counter-answer ratios, which that benchmark reports only qualitatively.

We will strip or neutralize chart titles in a controlled half of the data, following Plot Twist, so title shortcuts cannot confound results. We also re-render a "fresh chart" for each item with the update already drawn in. Accuracy on the fresh chart minus accuracy on stale chart plus update text defines the integration gap: the cost of having to merge external knowledge rather than read it off the image.

A smaller hand-curated slice will test whether the synthetic findings transfer to the messier case. It contains publicly revised statistics, where an agency later corrected a published figure. We will report it separately because it is small.

The ledger method. Given an open-weight backbone, the model first writes a ledger. Each row is a cell read from the chart, labelled with its visual anchor so that questions about "the orange line" remain answerable. This keeps what pure derendering throws away, as ChartQA found. The model then writes, for each update, the cells it supersedes or adds, or an explicit "applies to none" with a reason drawn from entity, period or date mismatch. A short program computes the answer over the merged ledger. A BeaconChart-style check re-queries the image for the cells the program actually used.

We train the ledger and scoping steps with supervised fine-tuning on StaleChart's training split, where gold ledgers are available for free from the tables. We also evaluate a prompted-only version on a closed model.

Baselines

The naive reference point is the backbone given the stale chart, the update text concatenated into the prompt, and an instruction to use the update. This is standard retrieval augmentation. Every method sees the same updates, so retrieval quality is held fixed. Beating it is necessary but not sufficient.

The strongest existing chart system is BeaconChart on the same Qwen3-VL-32B backbone, with the update text added to its planner's input. On original ChartQA it reports 91.60% accuracy, and on ChartX 72.05%, against an implied 63.45% for the base model, per BeaconChart. No reported result was found on Lacuna for any method on chart reasoning with external updates. Beating it means matching its accuracy on our unchanged-value questions while clearly reducing stale and grounding-free answers on update-dependent questions.

The strongest derendering baseline follows Chart-based Reasoning. It reports 81.3% ChartQA accuracy for ChartPaLI-5B with program-of-thought refinement. Our version derenders the chart to a plain table, lets a strong text model edit it with the update, and runs a program over the result. It uses the same backbone for derendering, and the edit model gets the same tuning budget.

The steering baseline ports ContextFocus to the multimodal backbone. That paper steers frozen text models toward retrieved context by contrasting activations with and without context plus instruction, and reports convergence with about 1.5k examples. We build its vectors from the same number of StaleChart training items. We add a perception-preserving variant inspired by Dynamic Multimodal Activation Steering, building the visual contrast from chart perturbations rather than object swaps from an object detector.

The fairness control is plain fine-tuning on the identical StaleChart training data with the same compute, supervising answers only. This tells us whether the ledger structure matters or whether the data alone does the work.

The key ablation keeps the ledger and program but removes the scoping step, pasting update text alongside the ledger. If this ablation matches the full method, our core claim is wrong.

Why not just…?

Why not just prompt the model to follow the update and think step by step? MATH-Vision reports that chain-of-thought sometimes lowers multimodal accuracy. MMKC-Bench reports that models which detect conflicts still answer from memory. Both predict a high stale-answer rate under prompting. Our naive baseline measures this directly, with and without chain-of-thought. If prompting alone closes the integration gap on a strong closed model, the problem is a weak-model artifact, and we would say so.

Why not just derender the chart, edit the table as text, and compute? This is the real threat. When the chart is clean and questions refer only to labels, we expect it to do nearly as well as our method, because it performs an implicit merge. Our case against it rests on two findings. ChartQA found that table-only pipelines fail on questions about visual attributes. ChartX's unusual chart types, such as radar, treemap and candlestick charts, are harder to derender. We will report results separately on visual-reference questions and on ChartX chart types. If derendering plus editing ties us everywhere, we will conclude that scoping can live in a text editor and the visual anchors are unnecessary. That would be a useful negative result.

Why not just steer activations toward the retrieved context? This is the obvious transfer of ContextFocus, and it may well reduce stale answers. Our concern is that it pushes in one direction only, toward trusting context. So it should raise over-application on distractor updates and grounding-free answers, which a single steering direction cannot distinguish from correct adoption. The distractor class in StaleChart is built to expose exactly this trade-off.

Why not just fine-tune on the update data end to end? If answer-only fine-tuning on the same data matches the ledger, the explicit structure is unnecessary. The matched-compute fine-tuning baseline settles this. The held-out-chart-type split tests whether either approach generalizes beyond the training distribution.

Proposed Experiments and Decisive Checks

We will run all methods on Qwen2.5-VL and InternVL3, the open families used in MMKC-Bench, plus prompted runs on a closed model. We report five quantities on update-dependent questions: merged-answer accuracy, stale-answer rate, update-only rate, over-application rate on distractors, and accuracy on unchanged-value questions. The last of these is the analogue of CARML's locality check. We also report the integration gap for each method.

The first decisive check is an oracle decomposition. Separately supply the gold ledger, then the gold scoping, then both, and measure how much of the integration gap each closes. If gold perception alone closes most of it, our claim that scoping dominates is false, and the work should become a perception paper.

The second check is the scoping ablation described above.

The third check compares the steering baseline against the ledger on distractor updates. It tests whether the field's default remedy for stubbornness trades it for gullibility in this setting.

The fourth check uses the title-stripped versus titled halves to measure whether update text acts as a new title-style shortcut, which we expect to inflate grounding-free answers under every method except the ledger.