Multimodal models can describe a chart fluently and still get the arithmetic wrong, or read the wrong line. Two failures stack. The model often cannot turn what it sees into a correct sequence of steps. It also misreads elementary details such as axis limits or which legend entry is which. We propose to fix both with an old tool from text mining: lexico-syntactic patterns, meaning recurring phrase shapes like "the difference between X and Y" that reliably signal a particular relation. Chart questions reuse a small number of such shapes. We would bootstrap a library that maps each shape to a short executable program with typed empty slots, such as "value of series S at category C." At test time we retrieve the right program by exact structural matching on the question, not by embedding similarity. The vision-language model is then asked only to fill the slots with small, checkable look-ups on the image, and a Python interpreter does the reasoning.
In two sentences: questions are routed by their syntactic shape to verified program skeletons, so the model never improvises the reasoning chain. The model is only asked elementary perceptual questions, and those are where its errors can be caught. The retrieval step borrows the constrained-decoding idea from text retrieval-augmented generation (RAG, where a model looks up external material before answering). What gets looked up here is a program, not a passage.
Core Claim
Our claim is about the human-written questions in ChartQA and the harder chart types in ChartX. On these, a program library that is bootstrapped from lexico-syntactic patterns and verified by execution, then retrieved by constrained translation, should beat two alternatives run on the same backbone and the same perception calls: free-form program-of-thought and dense retrieval of demonstrations. We expect the gain to come mostly from fewer cases where the right values are bound to the wrong things (wrong series, wrong axis, wrong operation), not from better pixel reading.
The evidence that motivates this comes from three places. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset attributes 42.2% of GPT-4V errors to reasoning rather than recognition, and finds that chain-of-thought prompting sometimes lowers accuracy. Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details shows that models do better on complex questions than on elementary reads, and that stripping the chart title drops accuracy from as high as 79% to the 30% range. That suggests the models are answering from text cues rather than grounded perception. Synthesize Step-by-Step shows that splitting questions into atomic sub-questions with Python code raises human-split ChartQA from 38.2% to 53.9%. Its limitation is that the decomposition exists only in offline synthetic training data built from SVG metadata. We move the decomposition to test time and make it retrievable.
Research Program
The pattern library. We start from a few dozen hand-written pairs of questions and programs over a small set of operators: lookup, argmax, difference, ratio, average, trend, and count-over-threshold. We parse the training questions from the ChartQA and PlotQA training splits with a dependency parser. We then mine the syntactic paths that connect the phrases filling each program slot. This is the bootstrapping loop of Mining and Re-ranking for Answering Biographical Queries on the Web and of the SPIKE-based pipeline in Identifying symptom etiologies using syntactic patterns and large language models, applied to questions instead of corpus sentences.
The etiology work needed about four hours of expert vetting to produce 60 patterns. We replace that vetting with an execution check. ChartQA training charts come with their underlying data tables, so we instantiate a candidate program, run it on the gold table, and keep the pattern only if the program reproduces the gold answer on a large enough fraction of the questions it matches. Questions that no pattern covers are handed to an LLM, which proposes a program. The program is verified the same way, and a new pattern is mined from it. The loop repeats until coverage stops growing.
The library is stored as a trie, a prefix tree that lists every valid pattern and slot type. From Retrieval to Translation (KG-Translator) supplies the method at this step: it translates queries into exact syntactic "hyperrelations" under trie and FM-index constraints, a text index that only allows substrings that actually exist. We reuse that constrained-decoding machinery. The target of translation changes from graph clues over a text corpus to program skeletons over a question library. This also avoids KG-Translator's known weakness with ordinary nouns, because our slots are typed chart roles (series, category, axis) rather than named entities.
Filling the slots. Each slot becomes an elementary question that the vision-language model answers on the image. An example is "What is the value of the orange line at 2015?" Because the question names the color and the series, the representation keeps the visual attributes that full table derendering loses. Chart-based Reasoning documents that its Chart2Table step fails on color-dependent questions for this reason.
Each slot value is cross-checked twice. First, it must be consistent with text read from the chart: the value must lie between the axis tick labels found by optical character recognition (OCR). Second, the model reads the value again on a crop of the relevant region. If the two checks disagree, the model reads the value once more. Unlike BeaconChart, whose verifier checks whole extractions and stores general-purpose natural-language rules, our checks run on individual typed values whose correctness is well defined.
Cold start. When no pattern matches, we fall back to free-form program-of-thought. We report pattern coverage separately so that gains cannot hide in the fallback.
Baselines
The naive reference point is the backbone answering directly, and then with plain chain-of-thought. We use Qwen3-VL-32B so that we can compare against BeaconChart on the same model. The file reports only relative figures for this backbone: BeaconChart's 72.05% on ChartX is stated as 8.6 points above the base model. We will rerun the base model ourselves.
The strongest existing method is BeaconChart. It reports 91.60% ChartQA accuracy and 72.05% ChartX accuracy with the Qwen3-VL-32B backbone. The file gives no human-versus-augmented breakdown for it, so we will rerun it under our evaluation split. Beating it means matching its overall ChartQA accuracy within noise while clearly exceeding it on the human-written split and on ChartX, at equal or lower inference cost measured in model calls.
The strongest structured-derendering method is ChartPaLI-5B with program-of-thought, at 81.3% on ChartQA in Chart-based Reasoning. We reproduce its recipe on our backbone as "DePLOT-style table, then LLM writes Python," because comparing across backbones would be unfair.
For demonstration retrieval we run MM-Retrieval, which does dense cross-modal retrieval of chain-of-thought examples, and the retrieval component of DCoT. Both are given our full verified library as their example pool. No reported result found on Lacuna for MM-Retrieval or DCoT on ChartQA; their reported numbers are on MathVista, ScienceQA, MM-Bench and MM-Vet.
Two ablations isolate the key ingredient. The first keeps the same library entries but retrieves them by dense embedding of the question, which tests whether structural matching matters. The second removes the library and lets the model write free-form programs that call the same slot-filling perception functions, which tests whether the verified skeleton matters. Every system gets the same training questions, the same backbone and the same tuning budget on a held-out slice of the ChartQA validation set.
Why not just…?
Why not just let the model write the program freely? This is the real threat. A strong LLM can write chart Python without a library, and program-of-thought already helps ChartPaLI. Our bet is that free-form programs fail in a specific way: they bind a correct operation to the wrong series or axis, and they silently skip perception. Verified skeletons with typed slots prevent both. The free-form ablation with identical perception tools settles this directly. If it matches our system on the human split, the contribution reduces to the slot-level perception checks, and we will report it that way.
Why not just retrieve demonstrations by embedding similarity? Dense demonstration retrieval is the approach of MM-Retrieval and DCoT. DCoT's own finding that question-text retrieval beats joint image-text retrieval suggests the useful signal for choosing a demonstration lies in the question's form. Two questions can be close in embedding space yet require different programs, for example "how many more" versus "how many times more." The dense-retrieval ablation over the same library measures exactly this. We expect the gap to concentrate on questions that are lexically similar but operationally different, and we will report accuracy on that stratum separately.
Why not just derender to a table and hand it to an LLM? Full derendering discards color and legend binding, according to Chart-based Reasoning. MSG-Chart found that structured visual-text edges beat flat serialization, lifting VL-T5 on human questions from 34.2% to 38.0%. Derendering remains a strong baseline on simple bar charts. We therefore stratify results by whether a question refers to a visual attribute, and we expect our advantage to appear only in that stratum.
Why not just use a bigger model? GPT-4o reaches 30.39% on MATH-Vision against 68.82% for humans. A 5B structured model already beats Gemini Ultra's 80.8% on ChartQA. Scale has not closed the gap on these tasks.
Proposed Experiments
The main evaluation uses relaxed accuracy (5% tolerance) on ChartQA, reported separately for the human-written and augmented splits, and accuracy on ChartX. The library is built only from ChartQA and PlotQA training data. ChartX therefore tests transfer to 18 chart types the library never saw, and we will report pattern coverage and accuracy on covered and uncovered questions separately.
The first decisive check is the free-form-program ablation described above. The second is title-stripped ChartQA, following the perturbation in Plot Twist. If slot-level perception is truly grounded, our accuracy should fall much less than the baselines when the title is removed. The third is Chart101, used as a direct measure of whether the slot cross-checks raise collective elementary accuracy, which Plot Twist reports is often below 25% even for top models. The fourth is an error audit of 300 failures, each labeled as a skeleton error, a slot-binding error or a pixel-reading error. It tests the central prediction that our gains come from fewer binding errors. If the audit instead shows the gains come from pixel reading, the lexico-syntactic library is not doing the work we claim.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Mining and Re-ranking for Answering Biographical Queries on the Web
- Identifying symptom etiologies using syntactic patterns and large language models
- From Retrieval to Translation: Translating Query into Graph-level Clues for Retrieval-Augmented Generation
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
- BeaconChart: Verifier-Guided Corrective Instruction for Chart Understanding
- Retrieval-Augmented Multi-Modal Chain-of-Thoughts Reasoning for Large Language Models
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- MSG-Chart: Multimodal Scene Graph for ChartQA