Vision-language models often answer chart and diagram questions correctly without having read the chart. Some take cues from the title; others describe the figure in safe generic terms such as "the values rise and then fall" and let the language model guess the rest. Accuracy on the final answer cannot tell these cases apart from real reading. Text-overlap metrics comparing a model's explanation to a reference cannot either, because they do not check which number goes with which label. We propose a scoring rule for chart reasoning, usable both as an evaluation and as a training reward, that gives credit only when three things hold together. The answer must be correct. The specific values the model claims to have read must be right. And the model must actually state the values the question depends on. The three parts are combined with a harmonic mean, an average that is dragged toward its smallest input, so strength on one part cannot buy back failure on another.
The two-sentence version is this. On charts produced from known data, every numeric reading in a model's reasoning can be checked by program against the source data, without asking another model for its opinion. Rewarding the harmonic mean of answer correctness, reading precision and reading coverage should penalize both vague, generic explanations and invented numbers. We test whether this closes the gap between "gets the answer" and "saw the evidence" better than rewarding the answer alone, adding a caption-consistency term, or asking a model to act as judge.
Core Claim
The claim is that generic output bias is one of the mechanisms behind the multimodal reasoning gap, and that it can be measured and trained against only with a reward that does not let one component compensate for another. Generic output bias is the tendency of captioning models to fall back on high-frequency, low-information phrasing; Discrete Mode Learning for Image Captioning and Measuring the Diversity of Automatic Image Descriptions document it in ordinary image captions. We conjecture that the same collapse happens in the perceptual steps of chain-of-thought reasoning, where it takes the form "a line graph showing an increasing trend" instead of "the 2019 point is at 42."
Two findings motivate the conjecture. Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details shows the "harder is easier" pattern: Gemini Pro Vision is about 10% less accurate on elementary readings than on complex ChartQA questions about the same chart. The same paper reports up to 79% accuracy when the title leaks the answer, falling to the 30% range when it does not. Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization attributes 51.3% of Qwen2.5-VL-7B's MathVista failures to perception.
The design point is specific. A reward for reading precision alone is maximized by saying nothing checkable, and generic output is exactly that. A reward for reading coverage alone is maximized by writing down many numbers, including invented ones. A reward for correctness alone is maximized by shortcuts. Each single signal can be satisfied through the failure that another signal prevents. That is why the aggregation has to be weakest-link and not additive.
Research Program
Checkable perceptual claims. The first component is a verifier. We render charts from programs whose underlying data are known: bar, line, scatter and grouped plots, following the template-and-code style of Synthesize Step-by-Step and the synthetic half of Chart101. Each question comes with the set of data points it logically depends on, and the generator records that set.
For each reasoning trace, a small extraction step pulls out atomic reading claims of the form (series, x-position or label, value). This step is plain information extraction; nothing is judged at this stage. Each claim is then checked numerically against the source data within Chart101's 5% tolerance. Checking the pairing of value, label and series is what n-gram metrics cannot do. Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning shows that CIDEr correlates with human judgment at only ρ = 0.542 on detailed captions and that attribute-binding errors pass unpenalized. A swapped pair such as "2019 = 35, 2020 = 42" has the same overlap with a reference as the correct version. Our verifier assigns it zero.
The score. For each trace we compute three quantities. $a \in \{0,1\}$ is answer correctness. $P$ is the fraction of stated readings that are correct. $R$ is the fraction of the question's required data points that are stated correctly. The score is
$$S = \frac{3}{\frac{1}{a+\epsilon} + \frac{1}{P+\epsilon} + \frac{1}{R+\epsilon}},$$
where $\epsilon$ is a small constant, for example 0.05. It keeps the score defined when a component is zero, which avoids the zero-score singularity of the plain harmonic mean. It also keeps a weak learning signal on wrong-answer traces that nonetheless read the chart well.
Each failure mode maps onto a component. A vague trace has $R \approx 0$, a hallucinating trace has low $P$, and a title shortcut gives $a = 1$ with low $R$. All three receive low $S$. At the dataset level we report mean $S$ together with "grounded accuracy," the fraction of items where $a = 1$, $P \geq 0.9$ and $R = 1$. The gap between plain accuracy and grounded accuracy directly measures how much apparent reasoning ability is not backed by reading.
This builds on two metric designs. HICEScore already takes a harmonic mean, but of global and local caption alignment, and it applies to natural-image captions with no reasoning step. CompreCap combines object, attribute and relation scores in a fixed arithmetic sum (0.25/0.35/0.40), under which strong coverage can mask failed relations. We move the weakest-link idea to reasoning traces, add answer correctness as one of the linked components, and replace model-judged alignment with numeric verification.
Audit. Before any training, we score open and proprietary-API models on the synthetic set, on Chart101, and on ChartQA charts whose source tables are available, in three title conditions: informative, removed and misleading. The prediction is that the gap between accuracy and grounded accuracy is large, and that it widens when the title is informative. That pattern would show that title shortcuts and generic reading are the same phenomenon viewed from two sides.
Training. We fine-tune Qwen2.5-VL-7B with Group Relative Policy Optimization (GRPO), a reinforcement learning method that scores each sampled response relative to others drawn for the same prompt. The reward is $S$, and training uses the DIVA-GRPO or MMR1 codebases. Because GRPO ranks responses within a group, the aggregation rule changes which responses count as better. For example, it decides whether a correct-but-vague trace outranks a wrong-but-well-read one. This is exactly where additive and harmonic rewards differ.
Baselines
The naive reference point is the untrained Qwen2.5-VL-7B, evaluated with and without chain-of-thought prompting. MATH-Vision found that chain-of-thought sometimes lowers multimodal accuracy, so both settings are needed. Published reference numbers are the following: Chart101 collective accuracy is below 25% for all top models, and the original ChartQA paper reports 45.52% for its best model, which is an older architecture. We found no reported result on Lacuna for Qwen2.5-VL-7B on Chart101.
The strongest existing method is CapPO from Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization. It adds a KL-divergence penalty, a measure of how far apart two probability distributions are, between reasoning conditioned on the image and reasoning conditioned on a detailed caption. It also reweights GRPO advantages by that consistency. It is the closest competitor because it targets the same perception-reasoning decoupling with the same base model and training method. It reports a +6.0% average gain across MathVista, MathVerse, MathVision, WeMath and DynaMath, and a 5.4% reduction in perception errors. We found no reported result on Lacuna for CapPO on ChartQA or Chart101. We will rerun it on our chart training set with captions built from the generator's data tables. It gets the same prompts, the same number of rollouts and gradient steps, and a hyperparameter sweep equal to ours. Beating it means higher grounded accuracy and a smaller accuracy drop when titles are removed, with plain accuracy that is equal or better on ChartQA and MathVista.
A third baseline is outcome-only GRPO, which rewards $a$ alone. It isolates whether any perceptual reward is needed. We also include Scene Graph Thinking (SaGe), which trains on structured scene-graph reasoning traces with additive node rewards. We use it as a check on whether structured supervision alone suffices, reproducing its additive reward over our verified components rather than its scene-graph pipeline.
The ablations isolate the key ingredient. Using the same three components, we compare the harmonic mean against a linear sum with weights tuned on held-out data, against a product, against a hard minimum, and against a gated reward that grants $P$ and $R$ only when $a = 1$. We also drop $R$, which leaves correctness plus precision. If the specificity argument is right, dropping $R$ should bring generic rationales back. The proposal's core claim holds only if the non-compensatory variants (harmonic, product, minimum) beat both the linear sum and the version without $R$. Whether harmonic specifically beats product is a secondary question that we will report honestly.
Why not just…?
Why not just reward the final answer and let reading emerge? Outcome reward cannot tell a title shortcut from a real reading, because both earn $a = 1$. Plot Twist's title manipulations show the shortcut is available and heavily used. The outcome-only baseline tests this directly. If it matches our method on grounded accuracy under the title-removed condition, then the specificity term adds nothing and we will say so.
Why not just use a multimodal model as a judge of the reasoning trace? Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge shows that such judges prefer fluent but visually wrong explanations, which it calls the plausibility trap. That paper's trained Perception-Judge-Flex-7B still misses perceptual errors 14.3% of the time (85.7% identification, versus 69.5% for base Qwen2.5-VL-7B). Used as a reward, that residual error is exactly what a policy would learn to exploit. This alternative is a real threat on real charts where no source data exist, so we include a judge-reward arm on the synthetic set. There we can measure how often it rewards traces that the program marks as wrong.
Why not just train the model to transcribe the chart into a data table first, then reason over it? This is the simplest serious alternative, in the spirit of Synthesize Step-by-Step. It may work well on simple synthetic charts. Its limitation is that it forces full transcription even when a question needs two values, and its errors compound: one misread cell corrupts every later step, the same pattern MATH-Vision reports for chain-of-thought. We run table-first supervised fine-tuning with the same data budget. If it matches our method on Chart101 and on ChartQA's human-written split, we will conclude that explicit structure, not the weakest-link reward, is doing the work. The remaining value would then be the audit metric itself.
Why not just use higher-resolution vision encoders? Chart101's failures occur on clean synthetic vector plots. GeoSQA finds that models still fail when diagrams are replaced by perfect text descriptions. Resolution does not address a model that chooses not to state what it sees, and our audit separates the two cases: low $P$ indicates misreading, while low $R$ indicates reluctance to commit to a reading.
Proposed Experiments
The first check validates the metric. On about 500 traces drawn from several models, annotators judge whether each rationale actually supports its answer with correct readings. We report Spearman correlation between their judgments and $S$, and compare it with CIDEr against reference rationales, with CLIPScore-style global similarity, and with the judge baseline. An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics reports that global reference-free metrics fail on hard negatives about 46% of the time. We therefore include swapped-value and swapped-label negatives and expect the numeric verifier to reject them almost always.
The decisive training check is the title-removal gap: plain accuracy with informative titles minus plain accuracy with removed titles, on Chart101 and on ChartQA. If the claim is right, our reward shrinks this gap relative to outcome-only GRPO and CapPO, and raises Chart101 collective accuracy above the sub-25% level reported for current models.
Three secondary checks follow. First, ChartQA's human-written split, where the original paper reports about 30% for models against about 61% on machine-written questions. Second, MathVista and MATH-Vision answer accuracy, to confirm the chart-trained reward does not damage general reasoning. Third, a hallucination audit tracking $P$ over training. The harmonic form should prevent the rise in invented detail that usually comes with pushing models away from generic output.
The main ways the claim could fail are clear. The linear-sum ablation might match the harmonic reward. The variant without $R$ might produce rationales just as specific. Or table-first fine-tuning might match everything. Any of these would show that the non-compensatory specificity argument is wrong, and we would report it as such.
References
- Learning Distinct and Representative Modes for Image Captioning
- Measuring the Diversity of Automatic Image Descriptions
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
- HICEScore: A Hierarchical Metric for Image Captioning Evaluation
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
- Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge for Mathematical and Scientific Reasoning Evaluation
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics