Rewarding Checkable Trends: Training Small Chart Describers Against the Data Table

Blind and low-vision readers usually meet a chart as an image with a one-line caption. That caption names the chart type and axes, but it rarely says which value is largest or which way a line moves. In an audit of alt text from human-computer interaction (HCI) papers, 90% of descriptions gave this construction-level detail, only 50% gave statistics, and only 31% described trends (Chintalapati et al.). Asking a vision-language model to fill the gap makes things worse in a different way. Across models, 82.06% of generated chart captions contain at least one factual error (Do LVLMs Understand Charts?), and trend statements on VisText are right only about half the time (VisText). The content screen-reader users most need is exactly the content models most often get wrong.

We propose training a compact vision-language model with reinforcement learning, where the reward comes from checking the model's own sentences against the chart's true data table. An LLM is used only to translate each sentence into a small formal claim, such as "the maximum of series A is at 2019" or "series B rises from 2010 to 2015." Ordinary code then decides whether each claim is true. The reward pays for true statistical and trend claims, weighted toward the facts that matter most in the table. It penalizes false claims, and it gives nothing for vague prose, so the model cannot score well by saying less. Tables are needed only during training. At test time the model reads pixels alone.

Core Claim

Prior work separates checking from learning. Inference-time filters prune unverified sentences after generation, and they leave thin summaries behind. Reinforcement-learning chart models reward question answers or table reconstruction instead of descriptive prose. Our claim is that the recurring analysis can be handed to deterministic code during training. That covers extrema, orderings, slopes, and comparisons. This turns open-ended description into a task with a dense, checkable reward. Optimizing that reward should raise the share of correct statistical (level 2) and trend (level 3) sentences, and it should do so without the loss of coverage that filtering causes.

The design borrows its central lesson from Think Fast, Talk Smart. That work kept numerical comparison and ranking in code while an LLM only phrased the results. Numeric error was 0.7% that way, against 16.9% when an LLM did the comparisons. We apply the same split to the reward function rather than to inference, and to chart images rather than sensor logs.

Research Program

The claim language. We first define a small typed grammar of chart claims. Level 1 claims cover chart type, axis labels, and axis ranges. Level 2 claims cover a value at a point, extrema, and averages. Level 3 claims cover the direction of a trend over an interval, a crossing between series, a ranking of categories, and relative size ("roughly twice"). The grammar also covers the error types catalogued in VisText's "six sins" and in the CHOCOLATE typology. Those include identity, value, direction, label, and magnitude errors.

Parsing and checking. A fixed LLM parser maps each generated sentence to zero or more claims in this grammar, or marks it unparseable. A deterministic checker evaluates each claim against the table. Numeric claims get tolerance bands, following the ±5% soft credit used in Chart-R1. Trend claims are checked against the sign and size of the fitted slope over the stated interval.

The salient-fact set. From each table, code also computes the facts a good description should mention: the global extrema, the dominant trend per series, the largest change, and crossings between series. This set serves as the coverage target.

The reward. For a generated description $y$ of chart $c$:

$$R(y,c)=\sum_{k\in T(y)} w_{\ell(k)}\, s(k) \;-\; \lambda\,|F(y)| \;+\; \mu\,\frac{|S(c)\cap T(y)|}{|S(c)|}$$

The symbols are defined as follows. $T(y)$ is the set of parsed claims the checker verifies as true. $F(y)$ is the set it finds false. $\ell(k)$ is the semantic level of claim $k$, and $w_{\ell}$ weights level 2 and 3 above level 1. $s(k)$ is a salience weight that discounts repeated or trivial claims. $S(c)$ is the salient-fact set.

Unparseable sentences earn nothing. We also cap their length so the model cannot pad with free text. Contextual level 4 content cannot be checked against a table, so it is excluded from the reward. Any external fact it states is penalized only if the checker detects it as an out-of-context number.

Training setup. We train Qwen2.5-VL-3B, the same base used by Chart-RVR, with Group Relative Policy Optimization (GRPO). GRPO is a reinforcement-learning method that scores several sampled outputs per input against each other instead of training a separate value model. Training starts from a supervised warm start. The training charts all come with true tables: VisText, the Statista split of Chart-to-Text, and code-rendered charts in the style of Chart-R1 that cover more chart types.

Auditing the checker before training. Before any policy optimization, we validate the parser-plus-checker against human error labels on CHOCOLATE. A reward that is wrong would be optimized as diligently as a correct one, so this step is mandatory.

Baselines

Naive reference. The do-nothing reference is the base Qwen2.5-VL-3B prompted for an accessible chart description. We also include the same model after supervised fine-tuning on the same training charts and captions. This gives "equal data, no reward" for comparison.

Strongest existing method. This is supervised fine-tuning on dense curated captions, as in ChartCap. ChartCap built 565K type-specific, hallucination-filtered chart captions. It showed that compact models fine-tuned on them beat Claude 3.5 Sonnet on its rendering-based Visual Consistency Score. That makes it the most direct evidence that curated data alone may suffice.

We will retrain the same 3B base on CHARTCAP data under the same compute and tuning budget as our method. The reinforcement-learning run's GPU-hours, spent instead as extra supervised epochs, define the matched budget. The file reports no factuality numbers for ChartCap on VisText or CHOCOLATE: no reported result found on Lacuna for ChartCap on these claim-level metrics. We will therefore measure it ourselves.

Beating ChartCap means a higher rate of verified level 2 and 3 claims per description, together with no lower salient-fact coverage. Both are measured on held-out VisText and Statista. We also require at least parity on Pew under independent judges.

Inference-time filtering. ChaTS-Pi samples several summaries, uses DePlot de-rendering plus PaLM-2 entailment to prune unverified sentences, and ranks what remains. It reported an 11-point faithfulness F1 gain on SciCap, and its critic correlates with human factuality judgments at Pearson above 0.50. We apply it on top of our supervised baseline with the same sample count used in our GRPO groups. We count its inference cost alongside its accuracy.

Training-data cleaning. Tackling Hallucinations in Neural Chart Summarization linearized tables and removed reference sentences whose BART-MNLI entailment score fell below 0.3. This raised human-judged value correctness from 56% to 76%. We reproduce that filtering on our captions before supervised training.

Ablations. Three ablations isolate the key ingredient, which is the deterministic checker. In the first, the claim checker is replaced by an LLM judge that scores whole descriptions, which removes the code-based checking. In the second, the coverage term is set to zero, giving $\mu=0$. In the third, the level weighting is made uniform.

Why not just…?

Why not just fine-tune on better captions, as ChartCap does? This is the real threat. If curated captions already carry correct trends, imitation may get most of the gain. The argument for reinforcement learning is that supervised training rewards resembling a reference, not being checkably right. The CHOCOLATE finding that fine-tuned models also make factual errors suggests imitation leaves errors behind. But that is suggestive, not decisive. The matched-budget comparison settles it. If ChartCap-style training matches our verified level 2 and 3 claim rate within noise, we will report reinforcement learning as unnecessary and recast the contribution as the checker used for evaluation.

Why not just use a strong LLM judge as the reward? Frontier judges show self-enhancement bias. GPT-4V inflated the factuality of its own corrected captions (Do LVLMs Understand Charts?). Judge prompts are also fragile, and a policy optimized against a fuzzy judge learns to please the judge. The LLM-judge ablation tests this directly. We expect the judge-rewarded model to score well on its own judge but worse under the code checker and in human audits.

Why not just filter at inference time, as ChaTS-Pi does? Filtering can only delete, so it trades coverage for faithfulness. NLI cleaning reduced nearly 30% of summaries to a single sentence, and VALOR-EVAL documents the same trade-off between faithfulness and coverage in general vision models (VALOR-EVAL). Filtering also inherits de-rendering errors and multiplies inference cost. Our comparison reports salient-fact coverage and latency next to accuracy, so a win on faithfulness that comes from deletion will be visible.

Why not de-render to a table and then generate deterministically? A pipeline in the style of Chart-RVR could do this: reconstruct the table, compute facts in code, and have an LLM verbalize them. It would inherit the clean numerics of Think Fast, Talk Smart. Its weakness is error cascades from de-rendering. The size of the gap between table and pixels on Chart-to-Text, 37.01 against 15.94 BLEU, shows how much is lost reading from pixels. We include this pipeline as an additional baseline, using the Chart-RVR model as the de-renderer. If it matches us on Statista, the honest conclusion is that what matters is the code-based computation, not the end-to-end learning.

Proposed Experiments

Checker validity. The first experiment asks whether the checker is valid, and it is decisive for everything after it. On CHOCOLATE, we measure agreement between parser-plus-checker error flags and human labels at the sentence level, broken down by error type. We compare against CHARTVE and ChaTS-Critic run on the same items.

Two failures would block training. One is a parser that misreads more than a small fraction of trend sentences. The other is a checker that agrees with humans less than ChaTS-Critic's reported correlation. In either case we would fix the grammar before any training. We also hand-audit 300 parsed claims from mid-training policy samples to detect reward hacking, such as sentences crafted to parse as trivially true claims.

Main comparison. The main comparison is evaluated on held-out VisText (whose L1–L3 annotations allow level-wise analysis) and Statista, which both have tables. There we report verified-claim precision per level, salient-fact coverage, and the share of descriptions with zero false claims. For contrast with the roughly 50% trend accuracy reported in VisText, we also report the trend-claim accuracy rate on its own.

We also report the level distribution found by the SciBERT level classifier of Chintalapati et al., which has 0.819 micro-F1. This directly tests whether the descriptions close the insight gap that alt-text audits documented.

Generalization without tables. Pew charts have no tables, so we score them with CHARTVE, ChaTS-Critic, and the rendering-based Visual Consistency Score from ChartCap. This keeps our own checker from grading its own model. BLEU-4 is reported only for continuity with ChartAdapter, which reported 35.55 on Pew. It is not a measure of faithfulness.

Human evaluation. Annotators blind to the system label every sentence in 200 descriptions per system for correctness and usefulness. If feasible, a small group of screen-reader users ranks descriptions of the same charts.

Decisive outcome. The proposal succeeds if the reinforcement-learning model beats matched-budget ChartCap-style training on verified level 2 and 3 precision at equal or higher coverage on both table-backed sets. The LLM-judge ablation must also lose under independent judges. If either condition fails, the result tells us that curated imitation or a soft judge suffices for this problem, and we would report that.