Reading Charts in Overlapping Tiles, Reconciled by a Conditional Random Field

Multimodal models often fail at the most basic step of chart questions: reading the numbers. On the Chart101 probe, leading models are less accurate at reading an axis minimum or estimating a coordinate than at answering compositional questions about the same chart. Fewer than a quarter of charts have all their elementary questions answered correctly (Plot Twist). A known cause is that the image is shrunk to a fixed resolution, which blurs small marks and labels. The obvious remedy is to cut the chart into overlapping crops ("tiles") and read each one at full resolution. That creates the problem long-document NLP already knows well. A tile in the middle of a bar chart sees bar tops but not the y-axis, so it cannot turn pixels into values. Marks near tile edges get read twice or cut in half. The partial readings then have to be stitched together, and today that is done with ad-hoc heuristics or by handing everything to a language model.

This proposal makes the stitching step a probabilistic structured prediction problem. A conditional random field (CRF) is a model that scores a whole joint assignment of labels at once, using per-item evidence plus terms that reward consistent combinations. Here, each tile reports candidate marks with pixel positions, colors, and readings of nearby tick labels. A CRF then jointly infers three things: the chart-wide mapping from pixels to values, which mark belongs to which x-category and which series, and which detections from overlapping tiles are the same mark. The reconstructed table is passed to a program-executing reasoner. The claim is that CRF reconciliation turns tiling from a resolution trick into a reliable de-rendering method. Its gains should appear specifically on elementary reading questions and on dense or large charts, where whole-image de-rendering breaks down.

Core Claim

Overlapping tiles fix the resolution problem but remove the global context needed to interpret what each tile sees. That context is mostly the axis calibration, the legend, and the left-to-right category order. In charts these are simple, nearly rigid structures, so they can be written as a small number of exact or soft constraints rather than learned implicitly by a generator.

The constraints have a natural structure. Detected marks, sorted by horizontal position, form a sequence. Labeling each mark with its x-category is a monotone segmentation, the same kind of problem as BIO tagging: category indices may stay the same or advance but never go backward, and that rule fits in a linear-chain transition matrix. Series assignment is a per-mark label tied to legend colors. The pixel-to-value mapping is a shared latent linear function, and every tile that can see tick labels contributes noisy evidence about it. Duplicate detections from overlapping tiles are handled with an explicit "same as previous mark" label whose score depends on geometric overlap.

We expect a CRF with exact inference over this structure to produce tables that are more accurate and more often structurally valid than either of two alternatives. The first is a generative model that writes the table autoregressively from the tiles. The second is heuristic merging such as non-maximum suppression, which keeps the highest-confidence detection and discards overlapping ones. Inference stays exact because the sequences are short and the axis mapping has only two parameters per axis.

Grounding in Prior Work

Three groups of prior work shape the design.

Work that motivates it. Plot Twist shows the perceptual failure and also the reliance on title cues: accuracy reached 79% when titles contained answers and fell to the 30% range without them. MATH-Vision attributes 31.9% of GPT-4V errors to recognition. Together these say that perception is a real bottleneck alongside reasoning. Our proposal addresses only the perception half and leaves reasoning to code execution.

Work that supplies the de-rendering-then-reason recipe. Chart-based Reasoning pre-trains ChartPaLI-5B to convert a whole chart into a table and adds Program-of-Thought, meaning the model writes code that is executed to get the answer. StructChart replaces flat CSV with (row, column, value) triplets. Both read the chart in a single pass and decode the table autoregressively. We change the reading step (tiled, at full resolution) and the assembly step (globally normalized inference), and keep their reasoning stage fixed. The original ChartQA work (ChartQA) used a ChartOCR pipeline with heuristic grouping. Our CRF is effectively a probabilistic replacement for that grouping. MSG-Chart builds visual and textual graphs over marks but uses them as features for message passing in a neural network; it does not decode a constrained table.

Work that supplies the mechanism. The overlap-and-reconcile pattern comes from text. Fixed-length windows with 50% overlap raised document-alignment F1 from 0.7880 to 0.9267 on MnRN (InstructCoder, as catalogued). Prepending a persistent global context header to every chunk helped summarization (doctor-patient summarization). We borrow the header idea directly: each tile is given the legend crop and the axis-title crop.

On the CRF side, linear-chain CRFs on clinical text and DeepVar show transition matrices removing invalid label sequences. Soft-Label Chain CRF trains a CRF over visual candidates with a soft target distribution when several candidates are acceptable, reaching 74.69% on Flickr30k Entities. We reuse that soft-label objective because several detections, for example duplicates from overlapping tiles, can legitimately match one gold mark. Bregman CRFs parallelize inference for long sequences. Our sequences are tens to hundreds of marks, so plain forward-backward inference suffices and we treat the Bregman variant as unnecessary unless we scale to very large figures.

Work that constrains the claim. Seq2Seq4ATE reduced illegal transitions to 0.02–0.03% with an autoregressive decoder and no CRF. That is direct evidence that a well-trained generator can learn structural validity, and it is why the generative merger is a primary baseline rather than a strawman. CRFIE warns that structured layers cannot recover candidates missed upstream. Our tile reader therefore over-generates candidates, and we measure recall before reconciliation separately.

Research Program

Tile reader. A chart reader is fine-tuned on synthetic tiles cut from SimChart9K (StructChart), for which exact geometry is available. Each tile is roughly 2x the backbone's native resolution, overlaps its neighbors by 50%, and carries the legend and axis-title header. For each tile, the reader outputs:

  • candidate marks, with pixel position and a color embedding;
  • OCR'd tick labels with their pixel positions;
  • an "edge-cut" flag for marks touching the tile border.

Reconciliation CRF. Let $m_1,\dots,m_n$ be the candidate marks pooled across tiles and sorted by horizontal position. Each mark receives a label $y_i = (c_i, s_i, d_i)$, where $c_i$ is its x-category, $s_i$ its series, and $d_i$ indicates whether it duplicates the previous mark. The chart also has a shared axis mapping $a = (\alpha, \beta)$, which converts pixel height $h$ to value $\alpha h + \beta$. The model scores

$$P(y, a \mid \text{tiles}) \propto \exp\Big(\sum_i \psi_e(y_i) + \sum_i \psi_t(y_i, y_{i+1}) + \sum_k \phi(a; \text{tick}_k)\Big),$$

where the three terms are as follows:

  • $\psi_e$ is the tile reader's evidence for a mark's category, series, and duplicate status.
  • $\psi_t$ forbids decreasing categories and scores duplicate links using pixel overlap.
  • $\phi$ is a robust (Student-t) fit of the axis mapping to each OCR'd tick label $\text{tick}_k$, so a misread tick label is down-weighted rather than allowed to corrupt the whole axis.

Inference. We alternate exact chain inference over $y$ with a closed-form robust update of $a$. Because the mapping has only two free parameters per axis, we can also enumerate a grid over it and take the maximum. The reader and the CRF are trained end-to-end with the soft-label objective of Soft-Label Chain CRF. The decoded table is serialized as triplets and handed to a fixed LLM that writes and executes Program-of-Thought code.

Analysis. We report table-cell accuracy under the 5% tolerance used by Chart101. We also report a structural-validity rate: the fraction of tables with no duplicated cell, no out-of-axis value, and no category-order violation. This mirrors the illegal-transition rate of Seq2Seq4ATE.

Baselines

Naive reference (whole-image Program-of-Thought). The same LLM receives a table produced by the same reader backbone applied to the whole downsampled image, with no tiles. This isolates whether tiling helps at all.

Strongest existing method: ChartPaLI-5B with Program-of-Thought. This is whole-image de-rendering followed by executed code. Chart-based Reasoning reports 81.3% accuracy on ChartQA. StructChart reports 78.5%, and Synthesize Step-by-Step reports 53.9% on human-written questions. If ChartPaLI weights are unavailable, we reproduce the recipe on our backbone with matched pre-training data and report both the cited and the reproduced numbers. Beating it means a statistically reliable gain on ChartQA-human and on Chart101 collective accuracy at no worse overall ChartQA accuracy. For Chart101 tiling, no reported result found on Lacuna; the cited reference is collective accuracy "often falling below 25%" (Plot Twist).

Generative merger. A model of the same size receives all tile outputs and is fine-tuned to emit the final table autoregressively, on the same data and with the same tuning budget as the CRF. This tests the Seq2Seq4ATE position that a generator learns structure implicitly.

Heuristic merger. Duplicates are removed by non-maximum suppression and each tile's values are calibrated by a RANSAC line fit to its own visible ticks, falling back to the nearest calibrated tile. RANSAC is a classical robust-fitting procedure that fits a line to random subsets of points and keeps the fit with the most agreement.

Ablations of the CRF. We remove each factor in turn: the shared axis mapping, the monotone transitions, and the duplicate label. The key ablation replaces the shared axis mapping with per-tile calibration. That substitution isolates the claim that global coupling is what tiling lacks.

All variants use identical tiles, reader weights, and reasoner, and receive the same number of hyperparameter trials.

Why not just…?

Why not just give a native high-resolution model the full image? This is the most serious threat. If the backbone can ingest the chart at full resolution, tiling may be unnecessary. The whole-image baseline at native resolution, run on dense PlotQA charts and on rescaled large charts, settles it. We expect the advantage to grow with mark count and image size and to vanish on simple ChartQA charts. If it does not grow, the method is not worth its tile cost and we will say so.

Why not just let an LLM merge the tiles? The generative merger is exactly this. The evidence that it could work is Seq2Seq4ATE. The evidence against it in the multimodal setting is the title shortcutting in Plot Twist and the consistency drift under chain-of-thought reported in MATH-Vision. We compare structural-validity rate and accuracy after title removal. A generator that is comparably valid and accurate would refute the need for the CRF.

Why not just fit the axis with classical geometry? The heuristic merger tests this. It should be competitive on clean synthetic plots. The CRF's expected edge comes from coupling three decisions: a misread tick label, a mis-assigned series, and a duplicated edge mark interact, and fitting each in isolation lets those errors compound. This is the error-cascade problem noted for ChartOCR-style pipelines in ChartQA.

Why not just use OCR'd text and a text model? GeoSQA found that even expert text renderings of diagrams left models near chance. MSG-Chart reports that adding visual graphs to OCR features raised UniChart from 82.36% to 85.36% on augmented questions. Values encoded only by bar height are absent from OCR, so text alone cannot recover them.

Proposed Experiments

Benchmarks. We evaluate on four sources:

  • ChartQA, overall and on the human-written split;
  • Chart101, for elementary reading and collective accuracy;
  • PlotQA, for dense synthetic plots, where the reference number is MatCha at 92.89% from Synthesize Step-by-Step;
  • a stress set of multi-panel and large scientific figures drawn from VeriSciQA.

Decisive checks. Three results would support the claim. First, gains should concentrate on Chart101 elementary questions and on charts with many marks, not on high-level ChartQA queries. Second, the drop in accuracy when titles are removed or altered, which Plot Twist puts at roughly 79% to the 30% range for current models, should shrink substantially, because answers come from reconstructed marks rather than text cues. Third, the ablation that replaces the shared axis mapping with per-tile calibration should account for most of the CRF's improvement over the heuristic merger.

Cost reporting. We report wall-clock time and reader calls per chart, since 50% overlap roughly doubles tile reads, as it does in text windows. We also report candidate recall before reconciliation, so that missed marks, which the CRF cannot recover, are kept separate from reconciliation errors. If tiled reading with CRF reconciliation fails to beat the high-resolution whole-image baseline on dense charts, the core claim is falsified.