Context-Bound Chart Derendering for Long-Document Question Answering

A chart in a report rarely explains itself. The legend says "HH-SR," and a paragraph three pages earlier defines it. The y-axis says "units," and a footnote states that values are in thousands. A methods section says that the dashed line is the baseline. When a long document is cut into chunks for retrieval, the chart and the sentences that give its marks meaning end up in different chunks. Current vision-language models (models that read images and text together) then fail in a specific way: they read the bars or points roughly correctly, but they attach the numbers to the wrong meaning.

We propose to fix this at the point where a chart is converted into a table. Derendering means turning a chart image back into the data table that produced it. Instead of derendering each chart in isolation, a model reads the chart together with every passage in the document that refers to it. It then writes a figure record: a table whose series names, units, and scope conditions are resolved from the text, where each resolved field points to the passage that justified it and to the image region it came from. Retrieval indexes these records, and arithmetic and comparison are done by executing a small program over the bound table, with the original image crop kept for questions about visual properties. Put plainly, the meaning that chunking cuts away from a chart is put back once, when the document is indexed, not at question time.

Core Claim

The claim is that a substantial share of the errors that remain in chart-dense document question answering, after retrieval has been made perfect, are binding errors: correct perception and valid arithmetic applied to a misinterpreted quantity. If that is true, resolving the binding during derendering should beat both isolated derendering and simply placing the relevant text next to the chart at answer time. Two pieces of evidence motivate the claim. First, Chart-MRAG Bench reports that Claude-3.5 Sonnet reaches only 58.19% correctness even when given ground-truth context, and it documents a text-over-visual bias. Second, CharXiv and MATH-Vision find that step-by-step reasoning amplifies early grounding mistakes instead of correcting them. The claim is falsifiable. If a carefully run "chart plus nearby text in the prompt" baseline matches our method under counterfactual edits to the defining text, binding at index time adds nothing, and we will report that.

Where This Sits

The intersection already has diagnostic and retrieval work, but nothing that binds definitions into a chart's symbolic form. Chart-MRAG Bench builds multi-hop text-chart questions and shows that unified embeddings reach 0% recall on chart-only retrieval and 22.32% Recall@5 on inter-document text-chart retrieval. It proposes no method, and we use it as our main testbed. IDentIfy encodes interleaved sections with a vision-language model and averages them for retrieval. URaG prunes whole pages inside the model after layer 6. Both improve which content reaches the reasoner, but neither changes what form a chart takes when it arrives. Anagent comes closest in spirit. Its expert agent fetches captions and citing text with parsers at question time, and it shows large gains from gold context. Our approach differs by doing this once per figure, offline, and by turning the fetched text into structured fields rather than more prompt text. VeriSciQA mines figure-citing paragraphs to build training questions. We reuse that pairing idea to build training data for the derenderer rather than for question answering.

On the method side, chart-to-table pretraining in ChartPaLI and decomposition into executable Python in Synthesize Step-by-Step give us the derender-then-execute structure. Both treat the chart as self-contained. EnSI-RAG shows that entity-level handles pointing back to source text beat whole-table units: 100% versus 91.5% on financial rows. Its handles cover only prose and tables, and figure records extend them to visual marks. Late Chunking and Landmark Embedding fix fragmentation for text by letting a chunk's representation see the whole document. We apply the same principle to charts, but the context enters as explicit bound fields rather than as a pooled vector.

Two results constrain the design. GeoSQA found that models still fail even after diagrams are perfectly transcribed into text, so binding alone cannot close the deduction gap. For that reason we hand deduction to an executed program rather than to free-form reasoning. Context-Aware Reasoner found that math-specialized reasoning models do worse on in-context reasoning. We read this as support for resolving context before deduction rather than hoping a stronger deductive model will recover it. ChartQA shows that table-only systems fail on questions such as "the orange line," which is why every figure record keeps its image crop and a mapping from each series to its color and region.

Research Program

The first step is a diagnostic that decides whether the rest is worth doing. On Chart-MRAG Bench, in the gold-retrieval setting, we take the failures of two strong models and have annotators label each one under a taxonomy modeled on MATH-Vision's. The labels are perception error (wrong value read), binding error (right value, wrong series, unit, scope, or definition, where the correct binding is stated elsewhere in the document), deduction error, and other. If binding errors are a small fraction, the proposal shrinks to a retrieval contribution, and we will say so.

The second step builds the context-bound derenderer. We fine-tune an open vision-language model; Qwen2.5-VL-7B is the base CAR used, which keeps comparisons consistent. Its input is a chart plus all passages that reference it, gathered by figure-number mentions and lexical overlap with legend and axis strings. Its output is a figure record: the data table, a resolved name and unit for each column, the source-passage identifier for each resolution, and a mark map from each series to its color, style, and bounding region. Training data comes from two sources. The first is synthetic: tables are rendered into charts with plotting code, legends are deliberately abbreviated or unit-scaled, and the definitions are placed in separate paragraphs at varying distances, so the correct bound record is known exactly. This follows the generator-driven approach of Synthesize Step-by-Step. The second is real scientific figures paired with their citing paragraphs, in the VeriSciQA style, with bound fields checked for consistency against the source text.

The third step indexes and answers. Records are embedded through their bound text fields and table header, so a query that uses the definition rather than the abbreviation can find the chart. This should address the chart-only recall failure. At question time the reasoner receives the retrieved records and image crops. It writes a Python program over the bound tables, which is executed, and may consult the mark map and crop for visual-reference questions.

Baselines

The naive reference point is a standard multimodal retrieval pipeline. Text is chunked, images are embedded with a CLIP-style model, and top-k neighbors go to the same 7B backbone. On chart-only retrieval in Chart-MRAG Bench this class reaches 0% recall, so any method should clear it easily. We include it to anchor the scale.

The strongest existing reference for answer quality is a frontier model given gold context. Claude-3.5 Sonnet reaches 58.19% correctness on Chart-MRAG Bench in this setting, as reported by that paper. Because that number uses perfect retrieval, we compare against it in our own gold-retrieval setting with identical inputs. Matching it with a 7B open model would already be notable, and exceeding it would count as beating it. For retrieval, the reference is the paper's 22.32% Recall@5 on inter-document text-chart questions.

The strongest method-level competitor is gold-context prompting in the manner of Anagent. The chart, its caption, and all citing paragraphs are placed in the prompt of the same backbone, which is fine-tuned on the same question-answer data and gets the same token budget as our records. No reported absolute result for AnaBench was found on Lacuna; only relative gains of up to 13.43% training-free and 42.12% fine-tuned are given. We therefore rerun this baseline ourselves rather than citing a number.

The key ablation is isolated derendering plus program execution, in the style of ChartPaLI and DePlot. The same derenderer is trained without the context input, and the citing text is supplied to the reasoner alongside the table. This isolates the single ingredient we claim matters: whether binding happens at derendering time. For reference, ChartPaLI-5B reports 77.3% on ChartQA, rising to 81.3% with program-of-thought. We use ChartQA to confirm that our derenderer does not lose single-chart skill relative to that line.

For long-document efficiency we compare against URaG's page pruning on SlideVQA and LongDocURL. URaG reports 92.9% top-1 retrieval on SlideVQA. All fine-tuned systems share the backbone, training examples, number of tuning runs, and context-token cap.

Why not just…?

Why not just put the caption and citing paragraphs in the prompt next to the chart? This is the real threat, because Anagent shows gold context helps a great deal. Our bet is that the model must still resolve "HH-SR" or "in thousands" while reasoning, and that the text-over-visual bias documented in Chart-MRAG Bench makes it rely on prose numbers or priors instead. The counterfactual-edit test below settles this directly. If prompted context tracks the edits as well as figure records do, our ingredient is unnecessary for answering, and the remaining contribution is retrieval and token cost.

Why not just convert every chart to a caption or table and run text retrieval? ChartQA shows that table-only systems cannot answer visual-reference questions. GeoSQA shows that perfect text transcription leaves deduction failures in place. The isolated-derendering ablation is exactly this alternative made as strong as we can make it, and the visual-reference subset of ChartQA tests the mark map.

Why not just feed the whole document to a long-context model? Chart-MRAG's 58.19% with gold context shows that having everything present is not enough. Landmark Embedding reports needle degradation with depth in 30K-token contexts. We run a full-document frontier model on documents that fit, and report accuracy against tokens processed.

Why not just use chain-of-thought or a math-tuned reasoner? MATH-Vision and CharXiv report that step-by-step prompting can lower accuracy by compounding early misreadings. Context-Aware Reasoner reports that Vision-R1-style training hurts contextual reasoning. We include a chain-of-thought variant of the prompting baseline so that this claim is checked in our setting rather than assumed.

Proposed Experiments and Decisive Checks

The primary benchmark is Chart-MRAG Bench, evaluated on correctness and Recall@5, split into single-hop, multi-hop, and inter-document text-chart questions. Its gold-retrieval mode separates reasoning gains from retrieval gains. Secondary evaluations are the CharXiv reasoning subset (GPT-4o reports 47.1% against 80.5% for humans), with each chart paired with its citing paragraphs from the source paper, and AnaBench. Chart101 and ChartQA serve as no-context controls, where our method should neither help nor hurt much. A drop there would signal that context-conditioning damages perception, which Plot Twist suggests is already fragile.

The decisive check is a counterfactual edit set built from Chart-MRAG and our synthetic documents. We change only the defining text, for example the unit scale, the expansion of a legend abbreviation, or which series is called the baseline, and recompute the gold answer. A method that truly uses context must change its answer accordingly. A method relying on priors or on text-embedded numbers will not. We report the fraction of edits whose answers flip correctly, for figure records, isolated derendering with the text supplied, and gold-context prompting. A companion check masks the chart image while keeping the text, to detect the "Clever Hans" shortcuts that Plot Twist describes.

The proposal succeeds if three things hold. Binding errors must form a sizable share of the diagnostic failures. Figure records must beat the gold-context prompting baseline on multi-hop correctness at equal token budget. And the gap must widen on the counterfactual set rather than vanish. If the prompting baseline ties on the counterfactual set, the binding claim fails, and we will report the method as a retrieval and efficiency result only.