Models that answer questions about charts often fail in a specific way. They can read the picture, and the language model inside them can do arithmetic, but the step that connects the two is unreliable. A model will copy a number that happens to sit in the chart title, or invent a value close to the right one, and then reason correctly from the wrong inputs. We propose a chart question-answering model that writes its reasoning as a short executable program. Every number in that program must be copied, by pointing, from text tokens actually present in the chart. The model is not allowed to type numbers from its vocabulary. Because the final answer is whatever the program computes, the model can never copy an answer straight out of the chart.
Pointing alone does not make the model point at the right place. It may simply match the question's words ("2018", "Germany") against identical strings in the chart's recognized text and grab whatever value sits next to them. So during training we also hide those matching "anchor" strings in the text input, at a moderate rate. The model then has to locate the right bar, line, or column from the image itself before pointing at its value. This borrows the idea behind correct-token masking from error correction, where hiding inputs that are already correct stops a model from passing them through unchanged. We redirect it to break the model's habit of matching words instead of looking at the chart.
Core Claim
Chart reasoning failures split into two parts. One is the value that enters the computation. The other is the computation itself. Current systems entangle the two because numbers are generated freely as part of the rationale. Our claim is that two changes together reduce both shortcut-copying and value hallucination more than either change alone, and more than cheaper substitutes such as constrained decoding or dropping titles at training time. The first change makes literal values a pointer-only channel inside program rationales. The second trains that pointer with anchor masking, so its targets are chosen by looking at the image.
The motivating evidence is consistent. MATH-Vision attributes 42.2% of GPT-4V errors to reasoning and 31.9% to visual recognition. Plot Twist shows that accuracy reaches up to 79% when the title contains the answer and falls to the 30% range when the title is removed or altered. That paper also shows that leading models answer fewer than 25% of elementary value-reading questions per chart collectively. ChartQA names transcription errors as a key bottleneck and observes that models copy isolated numbers on questions that require arithmetic.
Relation to Prior Work
The proposal reuses its methods from three places. From pointer-generator networks, introduced in Get To The Point as a learned gate that mixes vocabulary generation with copying from the source, we take the mixture itself. From FiD-PGN, which builds the copy distribution from existing decoder cross-attention and adds only 1,537 parameters, we take the implementation. That work is text-only open-domain QA, and its gate is learned freely. Ours is supervised by program syntax and points into chart text.
From program-style chart rationales we take the output format. Synthesize Step-by-Step (LAMENDA) produces question-to-rationale-to-answer data containing executable Python. ChartPaLI adds Program-of-Thought execution. Both let the model write literal values as ordinary generated tokens, and that is exactly the slot where hallucinated values enter.
From correct-token masking we take the training corruption. MaskCorrect hides already-correct source tokens and finds a narrow working range of about 15–20%. It also reports that merely zeroing the loss on correct tokens does nothing, because the inputs remain visible as a shortcut. PRINCE masks entity prefixes in the decoder so that entities must be retrieved from the encoder. It also shows that targeted masking beats random masking. We differ from both in what is masked. We hide input tokens that lexically match the question, never the pointer targets themselves, and the goal is to shift the model's grounding from string matching to the image.
CAAG is the closest multimodal use of masking. It masks the target word inside a draft-attention module for captioning. It has no pointing and no reasoning. MEAP shows that random context masking sharpens attention, raising Needle-in-a-Haystack retrieval from 52.8% to 85.8%. This motivates our expectation that masking will sharpen the very attention the pointer reuses, but MEAP has no pointer and no visual input.
LargePiG builds a training-free pointer from self-attention and a divergence-based copy probability. It has been tested on LLaVA, but not on reasoning chains, and it inherits whatever attention shortcuts the model already has.
Two papers constrain the design. Pointer-Generator Networks for Low-Resource MT: Don't Copy That! shows that an unconstrained copy gate copies when it should not. That is why our gate is tied to program syntax rather than left free. GeoSQA shows that perfect transcription of diagrams to text does not fix reasoning. That is why we do not claim pointing alone solves the gap. We target only the value-entry half and measure it separately.
Research Program
The model is an open vision-language backbone. Its input is the chart image plus the chart's recognized text, segmented into tokens: tick labels, legend entries, data labels, and the title. Its output is a program in the LAMENDA style, such as a = <ptr>; b = <ptr>; ans = a - b. At each decoding step $t$ the output distribution is
$$P_t(w) = g_t\,P^{vocab}_t(w) + (1-g_t)\sum_{i:\,o_i = w} \alpha_{t,i},$$
where $o_i$ is the $i$-th recognized-text token, $\alpha_{t,i}$ is the cross-attention weight on it (pooled as in FiD-PGN), and $g_t\in[0,1]$ is the generate-versus-copy gate.
The gate is supervised. In training programs, positions that hold numeric literals or category strings get target $g_t=0$. Operators, variable names, and syntax get $g_t=1$. At inference, literal slots may only be filled by copying.
One exception matters. Many charts have no printed data labels, so the true value is not among the text tokens. For those slots we allow a second literal type that is generated from the vocabulary as an explicit "estimate" token. We expect the gate to learn to use it on unlabeled charts. Without this, pointing would force wrong copies, the failure the Don't Copy That paper documents.
Training data come from ChartQA and PlotQA training splits plus LAMENDA-style synthesized rationales. Literal-to-source alignments are derived automatically by matching rationale values to recognized-text spans, and pairs that cannot be aligned get estimate labels.
Anchor masking is applied to a training copy of each example. Recognized-text tokens that also occur in the question or in the title are anchor candidates. A fraction $p$ of them is replaced with a mask symbol. Tokens that are pointer targets are never masked. The pixels are untouched, so the hidden year or category is still visible in the image, and the model must recover it from there.
We also mask title tokens that contain the gold answer, because that is the leak Plot Twist measures. Following MaskCorrect, we sweep $p$ over a 0–30% range. We then end with a short fine-tuning phase on unmasked data to close the gap between masked training inputs and clean test inputs.
Baselines
The naive reference point is the same backbone fine-tuned on the same rationale data to emit programs with ordinary generated literals, executed by the same Python interpreter. It gets no pointer and no masking. It is the setup of LAMENDA and of ChartPaLI's program variant reproduced on our backbone. Every other variant must beat it on ChartQA relaxed accuracy, on the human-authored split, and on literal accuracy to count as progress.
The strongest existing method on the leaderboard is ChartPaLI-5B with Program-of-Thought. Chart-based Reasoning reports 81.3% accuracy on ChartQA, and 77.3% without program execution. We cannot retrain a 5B PaLI model under matched conditions, so we report this number as an external reference rather than a controlled comparison. The controlled comparison of record is MatCha trained on LAMENDA data. Synthesize Step-by-Step reports 72.6% relaxed accuracy on ChartQA overall, 53.9% on human-authored questions, and 92.89% on PlotQA. If compute allows a MatCha-sized backbone, we rerun it with the same data, epochs, and hyperparameter search budget. Beating it means a statistically reliable gain on the human-authored split, where arithmetic and visual attributes dominate. We would claim little from the machine-generated split.
Two baselines test the alternatives directly. The first is constrained decoding: the naive model, with literal slots restricted at inference to strings present in the recognized text. It enforces "numbers come from the chart" without training a pointer. The second is the training-free LargePiG pointer applied to the naive model. It tests whether learned pointing adds anything beyond attention harvested from an untrained-for-pointing model.
Ablations isolate the key ingredient. We compare pointer without masking, masking without pointer, random-token masking at the same rate instead of anchor masking, loss masking on anchor tokens instead of input masking, and title-drop augmentation in place of anchor masking. All variants share data, steps, and an equal tuning budget of about eight configurations each, with three seeds.
Why not just…?
Why not just constrain decoding so numbers must appear in the chart text? This is the most serious threat, because it gets the "no invented values" property for free. It leaves untouched the question of which present value is chosen. The Plot Twist results suggest that choice is driven by text shortcuts such as the title rather than by the image. If constrained decoding matches the full system on literal accuracy and on the title-perturbation test, then the pointer is redundant. In that case the contribution reduces to anchor masking applied to a constrained decoder, and we would report it that way.
Why not just drop titles at random during training? This cheaply removes the specific leak Plot Twist measured. It does nothing about the more common shortcut of matching question words against axis and legend text. Anchor masking targets that shortcut. The comparison runs both at matched rates, and the decisive split is human-authored questions about color- or position-referenced series, where title dropping should not help.
Why not just derender the chart to a table and let a language model reason over it? ChartQA and Chart-based Reasoning both report that tables lose color and style cues, which hurts questions like "the orange line". GeoSQA shows that even perfect transcription does not fix reasoning. Our design keeps the pixels in the loop for locating the series and uses text only for exact value transfer. We report the color-referenced subset separately so this difference is visible.
Why not just scale the model? The scale evidence in the file points the other way. ChartPaLI-5B beats PaLIX-55B, 77.3% to 70.9%, and GPT-4o reaches only 30.39% on MATH-Vision against 68.82% for humans. Our method adds a gate of about a thousand parameters and a data corruption step, so its benefit, if any, should be orthogonal to scale. Running the method at two backbone sizes tests this.
Proposed Experiments
The main evaluation is ChartQA, reported separately for human-authored and machine-generated questions, plus PlotQA. Each comparison is run at matched settings against the baselines above. Three diagnostic checks decide whether the mechanism works as claimed rather than just raising a headline number.
The first check is literal accuracy. Where underlying data are available for alignment, we measure the fraction of program literals equal to the true chart value, separately for labeled and unlabeled charts. We also measure how often the model fills a literal slot with a number that appears in the question or title but is not the correct value. Pointing should cut hallucinated literals. Anchor masking should cut wrong-but-present literals. Each component should move its own error type, and the full model should move both.
The second check is shortcut robustness on Chart101 from Plot Twist. We test elementary value-reading questions under title removal, misleading titles, and added grid lines. We report the drop from the original to the perturbed condition, against Plot Twist's reported 79%-to-30%-range pattern for existing models. Success means a substantially smaller drop for anchor masking than for title dropping or constrained decoding.
The third check is gate behavior. The pointer gate should copy on labeled charts and switch to estimate tokens on unlabeled scatter and line charts. Plot Twist reports ChartLlama at 4.2% on scatter plots, so these are where forced copying would be most damaging. We report the masking-rate sweep to test whether MaskCorrect's 15–20% range transfers. We also report pointer attention sharpness, to test MEAP's prediction that masking concentrates the attention the pointer reuses.
The hypothesis is falsified if constrained decoding plus title-drop augmentation matches the full system on human-authored accuracy, literal accuracy, and perturbation robustness. As a transfer check, we evaluate on the chart-derived problems in MathVista without further training. There we expect gains only where value entry, not deduction, is the bottleneck.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- Get To The Point: Summarization with Pointer-Generator Networks
- A Copy-Augmented Generative Model for Open-Domain Question Answering
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
- Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction
- PRINCE: Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training
- Image Captioning with Context-Aware Auxiliary Guidance
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- Pointer-Generator Networks for Low-Resource Machine Translation: Don’t Copy That!
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts