Vision-language models, which answer questions about images, score well on chart question answering. Many of the charts in these benchmarks show real-world facts the model has already seen in text: populations, GDP, election results, company revenue. A model can get the answer right by remembering the fact without reading the chart at all. Benchmarks reward that shortcut because their gold labels agree with what the model remembers. The shortcut fails in the case that matters most once a model is deployed, when the chart shows data that is newer than the model's training. We propose to measure how much of current chart-reasoning accuracy is memory rather than reading. The probe is "counterfactual twins": real-entity charts re-rendered with values that are shifted, time-extended, or truly post-cutoff, where every answer is computed exactly from the underlying table. We then test whether a training signal built on these twin pairs teaches models to follow the pixels, including on combinations of question types and conflict conditions never seen in training.
The note combines four directions with unequal weight. The multimodal reasoning gap supplies the object of study. Knowledge obsolescence supplies the failure regime: the chart is newer than the model. Benchmark label noise supplies the evaluation discipline, since labels are generated by program and are never annotated. Combinatorial generalization supplies the test of whether the fix transfers to held-out compositions.
Core Claim
Our hypothesis is that on real-entity charts, a measurable share of end-to-end model accuracy comes from parametric memory, meaning facts stored in the weights. Under this hypothesis, accuracy should drop sharply when the chart contradicts memory, and the model's wrong answers should often echo the real-world value rather than being random misreadings.
We further expect two-stage "derender then compute" pipelines to be less exposed but not immune. Their extraction stage is itself a vision-language model and may quietly "correct" unusual values toward what it remembers.
Finally, we claim that training on paired twins, where the same entity and question appear with original and shifted values and the reward requires the answer to change with the chart, reduces this memory echo more than training on synthetic charts with random values. We further claim the gain holds for question-type by conflict combinations held out of training.
Why This Gap Is Real
Several papers motivate the hypothesis without testing it directly. Plot Twist probes models with elementary chart questions such as axis ranges and coordinates. It finds accuracy near 79% when the chart title contains the answer and in the 30% range when the title is removed or misleading. That is evidence of text shortcuts, but the paper does not separate title shortcuts from memory of the depicted facts.
Benchmarking Multimodal Knowledge Conflict (MMKC-Bench) builds conflicts between retrieved context and internal knowledge. It shows that models detect the conflict 75–79% of the time yet still answer from memory, and that Qwen2.5-VL-72B is more stubborn than the 3B model. Its conflicts are synthetic edits of entity knowledge in a retrieval setting. They do not involve numeric reading from a chart, and the paper leaves naturally occurring temporal drift unstudied. We move the conflict into the image itself and include real post-cutoff data.
Mitigating Temporal Misalignment by Discarding Outdated Facts models how long text facts stay valid and discounts confidence on stale ones. That paper is text-only. In our setting the fresh fact is already present in the input, and the question is whether the model defers to it.
The closest method work constrains the design. StructChart derenders charts into (row, column, value) triples before reasoning. Chart-based Reasoning pretrains on chart-to-table conversion and adds executable programs. Synthesize Step-by-Step decomposes questions into sub-questions and Python code. Chart-RVR rewards table reconstruction and process conformity with Group Relative Policy Optimization (GRPO), a reinforcement-learning method that scores a group of sampled answers against each other. All four evaluate on charts whose labels agree with the world. None asks whether its extraction stage is faithful when the chart and memory disagree, so none can tell reading apart from remembering.
On the label side, Re-TACRED and MMLU-Pro show that noisy or easy gold labels compress real differences between models. VeriSciQA filters scientific figure questions with model judges. Those judges are themselves vulnerable to the perceptual bias documented by Perception-Judge, in which judge models reward fluent rationales that misread the figure. Programmatic labels avoid judge noise entirely, which is why we use them.
Research Program
Twin benchmark. We first build a twin benchmark. We collect real statistical series about well-known and obscure entities and render each series in several chart styles. Each original chart is paired with twins of three kinds:
- Shifted: values perturbed enough to change the answer.
- Extended: the series continued past the model's training cutoff with plausible synthetic values.
- Natural drift: where available, the series continued with real post-cutoff data, so the fresh fact is genuinely new rather than counterfactual.
The original and its twins share renderer, style, and question. Only the data differ. Questions are built from primitive operations: lookup, extremum, difference, ratio, latest value, and trend over the last k points. Each is crossed with entity familiarity and conflict status. All answers are computed from the table, so the benchmark has no annotation noise ceiling by construction.
Metrics. The metrics are chosen to separate memory from misreading:
- Conflict gap: accuracy on originals minus accuracy on twins.
- Memory-echo rate: among wrong answers on twins, the fraction that match the real-world value within tolerance rather than any other value.
- Extraction faithfulness: for two-stage pipelines, the cell-level error between the extracted table and the true table, scored on twins versus originals with StructChart's SCRM metric (an edit-distance and relative-error score for extracted tables).
Twin-consistency training. The proposed fix starts from a 3B–7B open model. GRPO is applied with a reward that has three parts:
- answer correctness on the chart actually shown;
- a paired term that pays only when the model answers both the original and its twin correctly, which requires the answer to move with the data;
- Chart-RVR-style table reconstruction.
Compositional split. Training covers only some combinations. For example, conflict twins appear in training only for lookup and extremum questions, and natural-drift twins never appear at all. Evaluation covers all combinations. This tests whether evidence-following composes across operations or is learned one question type at a time.
Baselines
Naive end-to-end reference. The do-nothing reference is end-to-end question answering with the base models: Qwen2.5-VL at 3B, 7B, and 72B, matching the MMKC-Bench scale comparison, plus other open models where checkpoints are available. Each answers directly from the image. This baseline quantifies the problem. Because MMKC-Bench found stubbornness increasing with scale, it also tests whether the conflict gap grows with size.
Prompted reference. The same models receive an instruction to answer only from values visible in the chart and to ignore prior knowledge. This is run with identical decoding, and we report the best of three instruction phrasings.
Strongest existing method. The strongest competitor is the derender-then-compute family. The file reports 81.3% ChartQA accuracy for ChartPaLI-5B with Program-of-Thought refinement in Chart-based Reasoning, and 78.5% for StructChart's triples plus GPT-4 against 75.5% for end-to-end GPT-4V in StructChart. This family is the main threat because pushing all reasoning through an extracted table should, in principle, cut memory out.
ChartPaLI checkpoints may not be available, so we reproduce the recipe at matched scale. The open base model is fine-tuned for chart-to-table conversion and paired with a program executor. It uses the same training charts as our method and the same number of gradient steps.
Chart-RVR. Chart-RVR is the closest training-signal competitor. The file reports +7.28% out-of-distribution improvement over supervised fine-tuning on EvoChart, but no reported result found on Lacuna for any chart–memory conflict setting. We retrain it on our training pool with the same GRPO budget, number of samples per prompt, and hyperparameter search as our method.
Bar for success. Beating these baselines means a lower conflict gap and memory-echo rate on held-out combinations and on natural-drift twins. It also requires original-chart accuracy within one point of the best baseline and no loss on ChartQA.
Ablations. The first ablation is central. It replaces paired twins with an equal number of unpaired synthetic charts with random values, in the style of SimChart9K from StructChart, under the same reward and budget. This isolates the pairing as the key ingredient. A second ablation removes the table-reconstruction term to test whether faithful extraction drives the effect.
Why not just…?
Why not just prompt the model to ignore its prior knowledge? This is cheap, and it is in the baselines. MMKC-Bench suggests it will fall short: models already recognize the conflict three-quarters of the time and still answer from memory, so the failure is not a lack of awareness that instructions could fix. If prompting closes most of the conflict gap, that is itself a publishable negative result for the training method, and we will report it as such.
Why not just use a derender-then-compute pipeline? This is the real threat. If the extraction stage is faithful on twins, the pipeline solves the problem and our contribution becomes the diagnostic benchmark. The extraction-faithfulness metric settles this directly. Plot Twist's evidence that vision-language models lean on titles rather than reading values gives some reason to expect extractors to drift toward familiar values. We will not assume that outcome, however. We will report cell-level errors on twins versus originals.
Why not just train on random-value synthetic charts? Random values do break the link between the chart and memory, and StructChart shows synthetic charts are data-efficient. But random values on invented entities never place a familiar entity next to an unfamiliar number. That pairing is the condition under which memory competes with reading. The unpaired ablation tests exactly this claim. If it matches paired training, we drop the pairing claim and report that data composition alone suffices.
Why not just evaluate on existing benchmarks after cleaning their labels? Re-annotation in the style of Re-TACRED removes wrong labels. It cannot reveal memory reliance, though, because correct labels on real charts still agree with memory. Only charts whose data disagree with the world can separate the two.
Decisive Checks
The diagnosis stands or falls on the memory-echo rate. If wrong twin answers match the real-world value far more often than a uniform-misreading baseline predicts, memory is driving errors. The analysis of misreading should also reproduce Plot Twist's title effect inside our controlled renders.
Scale. Plotting the conflict gap against model size, from 3B to 72B, tests whether MMKC-Bench's scaling result transfers from retrieved text to pixels.
Composition and real drift. The held-out natural-drift set is the decisive test for the training claim. These twins never appear in training, and they reflect real temporal change rather than our perturbations.
ChartQA. Finally, we will re-render ChartQA charts where the underlying table is available, using both unperturbed and shifted versions. This estimates how much of reported ChartQA accuracy is attributable to memory. A large share would mean that the leaderboard spread reported for these benchmarks partly measures recall rather than reading.
References
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoning
- Re-TACRED: Addressing Shortcomings of the TACRED Dataset
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
- VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge for Mathematical and Scientific Reasoning Evaluation