Vision-language models, which read an image and answer questions about it in text, are now routinely asked whether a chart supports a claim. In English we already know they often reach the right verdict for the wrong reasons: their explanations describe bars, values and trends that are not in the chart. Nobody has measured what happens when the chart's title, axis labels and legend are written in Hausa, Amharic or Tamil. Speakers of those languages have the least access to human fact-checkers, so this is where the question matters most.
This note proposes a simple experimental lever. A chart is drawn from a data table by plotting code. We can therefore render the same table twice, once with English labels and once with labels in a low-resource language. The bars, lines and gridlines are identical in the two images, and the true answer to every numeric question is known exactly from the table. Any change in what the model says about the values between the two renderings is caused by the language alone. That paired rendering gives us a precise diagnostic of where low-resource chart understanding breaks. It also gives us a training signal that needs neither a frontier model writing text in a language it handles poorly nor a fallible automatic verifier.
Core Claim
We hypothesize that cross-lingual failures in chart verification are mainly failures to connect a legend or axis label to the marks it names. They are not failures to read the marks themselves. We further expect these failures to show up much more in explanation faithfulness than in verdict accuracy. When a model cannot read the labels, it falls back on what the title or claim suggests and writes a fluent explanation around that guess.
If this holds, two things follow. Measuring verdict accuracy alone will understate the damage in low-resource languages. The fix is also targeted: train the model so that its explanation stays the same under a change of label language, and check every number it states against the known table.
The contribution is threefold:
- a paired-rendering benchmark with exact ground truth for seven languages across five scripts;
- a small human-written test set of real charts in those languages;
- a training recipe built on cross-rendering consistency, with exact table checks used to build the preference pairs for direct preference optimization (DPO, a method that fine-tunes a model directly on pairs of preferred and rejected outputs).
What Exists and What Is Missing
Prior work that motivates the gap. The chart-integrity literature has established the faithfulness problem in English.
- ChartCheck verifies human-written claims against 1,700 Wikimedia charts. Its modular DePlot-DeBERTa pipeline reaches 73.8% accuracy against a 95.7% human ceiling. About half of GPT-4V's explanations contain errors even when the verdict is correct. The authors identify legends as a specific bottleneck.
- Beyond Answer Correctness formalizes the "faithfulness gap": the share of correct answers whose explanations contain claims the chart does not support. Their BakLLaVA base model gives correct and faithful outputs only 9.6% of the time. Their staged alignment pipeline cuts the gap from 87.1% to 35.7% while QA accuracy moves from 77% to 75.5%.
- ChartQA and ChartQA-X provide real charts with underlying tables and explanation-generation baselines.
All of these are English-only, and that is the gap we exploit.
Prior work that shows the cross-lingual collapse. xGQA shows that a visual question answering model trained in English drops from about 60% to about 20% accuracy when transferred zero-shot to seven other languages. It also shows that as few as 48 target-language images recover up to 20 points. M4FC shows that open-weight multimodal models such as Llama-3.2 degrade badly on Turkish and Tamil fact-checking. Both studies use photographs, not charts. Photographs let a model lean on coarse visual priors. Charts force exact binding of text labels to positions and scales, which is why we expect the cross-lingual damage to be worse and more concentrated in the explanations.
Prior work that constrains the design. Three results rule out some design choices.
- Dynamics of Instruction Fine-Tuning for Chinese LLMs finds that synthetic instruction data from frontier models plateaus earlier than human-curated data. This warns against generating low-resource training text with an English-biased teacher, which is how ChartCap builds its 565K captions and its code-based Visual Consistency Score.
- Beyond Answer Correctness proves that preference optimization cannot separate faithful from unfaithful outputs until the model already has basic competence in the domain. We therefore keep a supervised stage before any preference stage.
- The same paper's automatic verifier marked about 47% of claims "uncertain" because values had to be extracted from the image. Our rendered charts remove that problem: the table is known.
Prior work that supplies the method. Structure-Aware Pre-Training for Table-to-Text Generation shows that structure-aware objectives on tables reduce hallucinated table facts. We borrow its idea of training against the table's structure, applied to rendered images rather than linearized text. The low-resource literature contributes two observations. Multilingual tokenizers split such languages into many fragments (Efficient Low-Resource Language Models Using Tokenizer Transfer; ViT5). Training over related languages together helps (AfriSenti-SemEval). We use both to choose languages and to test whether fragmentation of label text predicts binding errors.
Research Program
Paired-rendering benchmark
The first artifact is a rendered benchmark.
- Tables. We draw from ChartQA, whose real charts come with underlying data, and from a filtered subset of the WDC web tables used by STTP.
- Rendering. Each table is drawn as a bar, line, pie or scatter chart with matplotlib and Vega-Lite. Every chart is produced in English and in seven target languages: Indonesian, Turkish, Bengali, Tamil, Amharic, Hausa and Yoruba. These span Latin, Bengali, Tamil and Ge'ez scripts, including Yoruba's tone diacritics. Labels are translated by machine and corrected by native speakers. Labels are short strings, so correction is cheap, and a translation error never corrupts the ground truth, because the truth lives in the numbers.
- Label conditions. Each table is rendered under four label conditions with identical marks:
- English labels;
- target-language labels;
- neutral placeholder labels (A, B, C);
- target-language labels that are directly annotated on the marks, with no separate legend.
- Claims. Native speakers write about forty claim templates per language, covering comparison, extremum, trend, difference and proportion. Values are filled in from the table, so claim text involves no frontier-model generation.
Faithfulness scoring against the known table
Every numeric or ordinal statement in a model's explanation is parsed and checked against the table. Numbers must match within half a gridline interval, and orderings must match exactly. Because the table is known, the "uncertain" category is close to empty by construction. We report:
- verdict accuracy;
- the correct-and-faithful rate;
- the faithfulness gap as defined by Dahiya and Saini;
- a legend-binding error rate: the fraction of unsupported statements that attribute a value to the wrong series or category.
The comparison across conditions is the core diagnostic. Our attribution test asks whether target-language behavior looks like English behavior or like placeholder behavior. If it looks like the placeholder condition, the model is effectively not reading the labels.
Real-chart test set
Rendered charts are cleaner than the real thing, so the decisive evaluation uses real charts. We collect about 300 charts per language from national statistics offices, news outlets and Wikipedia editions in those languages. For each chart, two native speakers write supported and refuted claims with explanations, following the ChartCheck protocol. Explanation faithfulness on this set is judged by humans. This set is never used for training.
Training recipe
The method runs in two stages on an open-weight model (a Qwen-VL-family model at roughly 7B parameters).
- Competence stage. Supervised fine-tuning on paired renderings. Targets are template explanations whose numbers come from the table.
- Preference stage. For each table, the model generates explanations under the English and target-language renderings. An explanation is preferred when all its stated values are supported by the table. It is rejected when it states values that the table contradicts, especially values that differ from what the same model said under the English rendering.
We define the cross-rendering disagreement for table $T$ and language $\ell$ as the fraction of value statements in the target-language explanation that contradict $T$, minus the same fraction for the English rendering. Driving this to zero is the training goal, and it is also the headline diagnostic. No step requires an automatic natural-language verifier or a frontier model writing target-language text.
Baselines
Do nothing. The naive reference point is the untouched open-weight VLM, prompted with chain-of-thought in the claim's language. It establishes the size of the cross-lingual faithfulness gap. To situate it, we run the same model in English on ChartCheck and compare against the reported 73.8% DePlot-DeBERTa accuracy from ChartCheck. No reported result was found on Lacuna for chart fact-checking or chart faithfulness in any non-English language. Our real-chart set would be the first such number.
Translate-test. This is the strongest practical competitor. We follow the modular pipeline that won on ChartCheck: DePlot extracts a table from the target-language chart, the extracted cell strings and the claim are machine-translated into English, and a DeBERTa verifier fine-tuned on ChartCheck decides. Its explanations are generated from the translated table. It is strong because, in English, modular extraction beat end-to-end VLMs, and translation sidesteps the target language entirely. For fairness, it gets the same translation system and the same native-speaker label corrections that we use for rendering.
Strongest existing faithfulness method. We run the Progressive Domain Alignment pipeline of Beyond Answer Correctness on the same base model and the same rendered tables, using its original setup: an automatic extract-then-verify judge for the preference stage. Its reported English result is a faithfulness gap reduced from 87.1% to 35.7% at 75.5% QA accuracy. This baseline isolates whether exact table grounding plus cross-rendering pairing beats a model-based verifier when everything else is held fixed.
Few-shot target supervision. Following xGQA, we fine-tune on a small human-written set per language with the same annotation budget our templates cost. In xGQA, 48 images recovered up to 20 points.
Ablation that isolates the key ingredient. The main ablation uses the same rendered charts and the same compute but plain supervised fine-tuning, with no cross-rendering preference stage. This separates "more multilingual chart data" from "consistency across label languages as the signal." A second ablation replaces re-rendered target-language charts with English charts plus machine-translated claims (translate-train).
Fairness and the bar to clear. All learned variants receive the same base model, the same number of training tokens, and the same hyperparameter search of eight configurations, selected on a rendered validation split. Beating the field means a higher correct-and-faithful rate on the real-chart test set than both translate-test and the Progressive Domain Alignment baseline, with verdict accuracy no more than two points lower, averaged over languages and holding on at least five of seven.
Why not just…?
Why not just translate everything into English first? This is a real threat, and the experiments are designed to settle it. The pipeline has a weak joint: the claim and the chart labels are translated separately. A Hausa claim naming a region and the Hausa legend entry for that region can come out as two different English strings, and the verifier then misbinds them. The weakness is sharpest for scripts that DePlot has rarely seen, where extraction itself fails before translation begins. We expect translate-test to be competitive on Indonesian and Turkish and to fall behind on Amharic, Tamil and Yoruba. If it matches our method on all scripts, the right conclusion is that low-resource chart verification is a translation and extraction problem, and the paper should say so.
Why not just fine-tune on machine-translated ChartQA and ChartCheck? Translating the questions leaves the chart labels in English, so the model never practices binding target-language labels to marks, which is the failure we hypothesize. Translating with a frontier model also invites the early plateau reported for synthetic data in Dynamics of Instruction Fine-Tuning. The translate-train ablation measures this directly against re-rendered charts.
Why not just collect a few dozen real examples per language, as xGQA did? It is cheap and sometimes effective, so we run it at a matched annotation budget. Small real sets cannot easily teach faithfulness, though: without known tables, nobody can check every number in a training explanation. If few-shot supervision matches us on the correct-and-faithful rate, our extra machinery is not worth it.
Why not just retrain DePlot on the rendered multilingual charts? This variant is a strong version of translate-test. It may also capture much of our gain, because better extraction fixes binding upstream. We include it in the translate-test arm, and we additionally apply our preference stage to the modular pipeline's explanation generator. That tests whether the consistency signal adds anything once extraction is good.
Proposed Experiments and Decisive Checks
The first decisive check is diagnostic, on the rendered benchmark with an untuned model.
- Falsification of the core claim. If the drop in correct-and-faithful rate from English to target-language labels is not larger than the drop in verdict accuracy, the claim is false.
- Binding versus mark reading. If target-language performance is not closer to the placeholder condition than to English, the failure is not label binding. Directly annotating values on the marks should then close most of the gap. If it does not, the problem is mark reading, and the method should target the image encoder instead.
- Tokenizer fragmentation. We regress the per-language binding error rate on the average number of tokens per label word. This tests whether fragmentation predicts binding failure, which no prior work has measured for charts.
The second check is the training comparison on the real-chart test set against all baselines above. We report the correct-and-faithful rate, verdict accuracy and human-judged legend-binding errors, with bootstrap confidence intervals over charts.
The third check is transfer. We train with renderings in four languages (Indonesian, Bengali, Hausa, Amharic) and test on Turkish, Tamil and Yoruba with no target-language training data. This asks whether consistency across label languages is a general skill or must be taught per language. The multilingual gains reported in AfriSenti-SemEval suggest transfer should be positive within language families.
Finally, we run English ChartCheck and ChartQA after training. This confirms that the method does not buy low-resource faithfulness by degrading English, measured against the English accuracy figures reported in ChartCheck and Beyond Answer Correctness.
References
- ChartCheck: Explainable Fact-Checking over Real-World Chart Images
- Beyond Answer Correctness: Measuring and Reducing Explanation Faithfulness Gaps in Chart Understanding VLMs
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- ChartQA-X: Generating Explanations for Visual Chart Reasoning
- xGQA: Cross-Lingual Visual Question Answering
- M4FC: a Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset
- Dynamics of Instruction Fine-Tuning for Chinese Large Language Models
- ChartCap: Mitigating Hallucination of Dense Chart Captioning
- Structure-Aware Pre-Training for Table-to-Text Generation
- Efficient Low-Resource Language Models Using Tokenizer Transfer
- ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation
- SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)