Where Chart-to-Code Models Lose the Digits: Separating Seeing Errors from Spelling Errors

When a vision-language model turns a chart or webpage screenshot into plotting or HTML code, many of its visible mistakes sit in short literal strings. It writes a bar height of 54.2 where the image shows 52.4, the hex color #FF5373 instead of #FF5733, or an axis label with a dropped character. The field treats all of these as visual hallucination, meaning output not supported by the image, and attacks them on the perception side with higher-resolution encoders, step-by-step extraction, and render-and-revise loops. There is a second possible culprit. The language model writes code in subword tokens, which are integer IDs for chunks of characters chosen by a compression heuristic. These IDs do not encode which characters they contain, so a model can know the right value and still spell it wrong when it has to emit it as a chain of opaque chunks.

This note proposes a measurement that splits literal errors in visual-to-code output into two kinds. A "seeing error" means the model's internal belief about the value is wrong. A "spelling error" means the model's belief is right but its token-level emission is wrong. The test is cheap and needs no retraining. For each wrong literal, we score the correct string by summing probability over every token sequence that spells it, not only the tokenizer's default one, and we check whether the model actually preferred the correct value. If spelling errors account for a meaningful share, a small decoding change follows directly: switch to character-level scoring only inside literals, using the code's own lexer to detect literal boundaries. That change should fix those errors without touching the vision encoder. If spelling errors are rare, the result tells the field to stop considering tokenizer fixes for this task and to spend its effort on perception.

Core Claim

We claim that some of the numeric, color, and text-literal errors that open-weight chart-to-code and UI-to-code models make are emission errors caused by subword segmentation, not perception errors. We claim this share can be measured directly by comparing character-marginalized likelihoods of the gold and emitted literals under the same visual context. We further claim that a decoder which marginalizes over token segmentations only inside lexer-identified literal spans will lower literal error rates on ChartMimic and Design2Code. It should do so without the execution-rate losses that whole-sequence constraints cause.

The motivation comes from two literatures that have not been joined. The chart-to-code benchmarks document literal failures but treat the model as a black box. ChartMimic catalogs invented labels and misplaced small data points without asking where they come from. Plot2Code names perception of fine annotations as the main bottleneck. The tokenization literature shows character blindness that scale does not fix. Tokenization Falling Short reports degradation that grows with string length. ByteFlow reports Llama 3.1 at 3.6% on CUTE spelling-inverse against 95.1% for a 1.3B byte-level model. Neither literature has tested the other's explanation.

The method comes from From Language Models over Tokens to Language Models over Characters. That paper sums probability over the token sequences that cover a target string, prunes the sum with a beam, and shows that canonical tokenization leaves real probability mass on the table: about 0.93 versus 0.76 bits per byte for DeepSeek-R1-Distill-Llama-8B on Wikitext-103. It evaluated only compression of static text, not generation, and not code. Code literals are a natural place to apply it, because the text just before a literal (color='#, width:) is exactly the kind of token boundary that paper shows misleads models.

Research Program

The first component is a literal-level evaluation layer for existing benchmarks. On ChartMimic Direct Mimic, we run the gold and generated scripts through ChartMimic's own code tracer and align the executed elements: text, colors, and data arrays. This yields aligned pairs of a gold literal $s^*$ and an emitted literal $\hat{s}$. Each pair is tagged by type: numeric data, tick or label text, hex or named color, and layout number. Each is also tagged as printed (legible in the image, so recoverable by text recognition) or inferred (for example, a bar height read off its geometry). For Design2Code we align text blocks and CSS color and size values the same way. This layer matters on its own, because the file contains no reported literal-level accuracy for any model on these benchmarks.

The second component is the attribution test. Fix the image and the generated code prefix up to the literal. Let $P_{\text{char}}(s)$ be the probability that the model continues with character string $s$, summed over all token sequences that spell $s$ and then a closing delimiter:

$$P_{\text{char}}(s \mid \text{image}, \text{prefix}) = \sum_{t \,\in\, \text{Cover}(s)} P(t \mid \text{image}, \text{prefix})$$

Here $\text{Cover}(s)$ is the set of token sequences whose concatenation is $s$. We approximate the sum with the beam-pruned covering enumeration from Vieira et al., at the beam size of about 8 where they report the approximation error flattens.

A wrong literal counts as emission-attributable when $P_{\text{char}}(s^*) > P_{\text{char}}(\hat{s})$, meaning the model, once freed from its canonical segmentation, prefers the right answer. It counts as perception-attributable when the gold string stays below the emitted one, or falls outside the top few strings found by a character-level beam.

We add two controls. The first is a pure-copy probe: the gold literal is given in the text prompt and the model is asked to place it into code. Any remaining error is spelling with no seeing involved. The second is a natural-language probe: we ask, for example, "what is the value of the third bar?" outside code context. This shows whether the value is available to the model when it is not writing code.

The third component is the intervention, which we call literal-span marginalized decoding. A lightweight lexer for Python and CSS tracks the decoder's position, in the spirit of the detokenizer-lexer composition in GREATGRAMMA. When the decoder enters a numeric, hex, or string literal, it stops sampling the next subword token. Instead it selects the literal by character-level beam search over $P_{\text{char}}$, which ends when the lexer sees the literal close. It then re-tokenizes the chosen string canonically and resumes ordinary decoding. Boilerplate code is never touched, so the distortion that Grammar-Aligned Decoding documents for whole-sequence masking has little room to arise. That paper reports a 38% geomean gain in correct solutions over naive grammar masking on INV-BV, and we treat its result as a constraint on our design rather than a competitor.

As a training-side comparison, we also run a short continued fine-tuning of the same backbone on Chart2Code-160k from ChartCoder under randomized segmentations, following StochasTok. That work reports retrofitting GPT-2 with character-level skill in 2,000 to 7,000 steps.

We use open-weight models whose likelihoods we can read: ChartCoder (a DeepSeek Coder 6.7B backbone) and InternVL2 for charts, plus an open UI model for Design2Code. Because backbones split digit strings differently, we record each model's digit and hex segmentation scheme. Comparing models gives a natural experiment: if spelling errors drive literal failures, their share should track how much each tokenizer fragments numbers.

Baselines

The naive reference point is each model decoded greedily, exactly as its paper reports, scored with our literal-level metrics alongside the standard ones. For context, ChartMimic reports GPT-4o at 81.2 on Direct Mimic with 93.2% execution, and InternVL2-Llama3-76B at 58.5 with 83.2% execution. ChartCoder reports 91.4% execution and a visual score of 74.0 on ChartMimic. Design2Code reports GPT-4o Block-Match falling from 93.0 to 56.6 on its HARD split. These are the targets that the literal-level analysis sits beneath.

The strongest existing open method on charts is ChartCoder with its Snippet-of-Thought prompting. That prompting first extracts layout, then data and colors, then style, before writing code, and it is the most direct perception-side attack on literal errors. We run it on the same images, with the same decoding budget and the same seeds as our intervention, and apply our decoder on top of it. Beating it means a statistically reliable drop in literal error rate on numeric and color literals, with no loss in execution rate and no loss in ChartMimic's text and color tracer scores. On Design2Code-HARD, UI2Code^N reports 88.6% against 41.9% for Qwen2.5-VL-72B. Its render-and-refine loop is the strongest competitor there. We compare against its single-turn and multi-turn modes at matched wall-clock budget, counting our extra literal-span scoring cost honestly.

The key ablation applies the same character-level beam to random non-literal spans of equal total length instead of to literals. This separates "marginalization over literals helps" from "any extra search helps." A second ablation adds plain grammar-constrained decoding with no marginalization. This separates syntactic repair from literal repair. No reported result was found on Lacuna for character-marginalized decoding on any code or visual-to-code benchmark, so these comparisons are new measurements.

Why not just…?

Why not just feed text extracted by an optical character reader into the prompt? This is the most serious threat for printed labels and tick text, and we treat it as a baseline, not a straw man. Design2Code found that adding extracted text improved text recall but left layout and color errors. Plot2Code found that extra textual conditions sometimes lowered pass rates. Neither result shows whether supplied strings are copied into code correctly. Our pure-copy probe tests exactly that. If copying is near-perfect, text recognition dominates for printed literals, and our contribution narrows to inferred data values and colors, which text recognition cannot supply. Splitting literals into printed and inferred makes this outcome visible rather than hiding it in an average.

Why not just ask the model to write numbers digit by digit, or with separators, and clean up afterward? This is a prompt-level way to sidestep segmentation, and it may work as well as our decoder. We include it as a baseline, along with the training-side StochasTok fine-tune. If either matches literal-span marginalization, we will recommend the cheaper fix. The attribution measurement still stands either way, because it is what justifies using any segmentation-aware fix at all.

Why not just use higher resolution, step-by-step extraction, or self-revision? These target seeing errors, and ChartCoder's high-resolution patching and its staged prompting do help. The attribution test predicts that they leave spelling errors untouched. We check this by measuring the emission-attributable error share before and after Snippet-of-Thought prompting. If staged prompting also removes most spelling errors, the tokenizer story weakens accordingly.

Why not just scale the model? Tokenization Falling Short and the ByteFlow comparison with Llama 3.1 argue that character blindness persists with scale. We still measure the emission-attributable share across model sizes within one family where weights are available, rather than assume it.

Decisive Checks

The program hinges on one number: the fraction of wrong literals on ChartMimic Direct Mimic and Design2Code-HARD for which the character-marginalized likelihood prefers the gold string. If that fraction is small, say under a tenth of errors, across backbones, the tokenizer hypothesis is largely false for this task. We would publish the literal-level benchmark and the negative result.

If the fraction is substantial, three further checks follow. First, literal-span marginalized decoding should recover roughly that fraction of errors, measured as exact match on numeric and color literals and as ChartMimic text and color tracer scores, with execution rate unchanged. Second, the random-span ablation should recover almost none of it. Third, gains should be larger for inferred numeric literals and hex colors than for printed text that text recognition already supplies, and larger on backbones whose tokenizers fragment digits more.

A secondary check targets the oscillation UI2Code^N reports during multi-turn polishing. It asks whether edits to small CSS numbers (14px to 15px) stabilize when those literals are chosen by character-level scoring, measured as the rate of reverted edits across refinement turns on UIPolish-bench.