Spending Verification Where Charts Break: Cost-Aware Search for Text-to-Visualization

A chart in a PDF or printed report cannot be questioned. A reader cannot hover over a clipped legend, re-sort a bar order, or check whether an axis starts at zero. So when a language model writes plotting code for a document, every defect it leaves becomes permanent. Current text-to-visualization systems try to catch these defects with a vision-language model (VLM), a model that looks at the rendered image and judges it. But they spend the same effort on every request. A one-series bar chart and a 3×3 grid of overlaid density plots get the same number of samples and the same critic calls. The adaptive-compute literature shows that spending compute evenly like this wastes it on easy inputs and starves hard ones. That literature, however, assumes verification is a single cheap or single expensive check. Chart verification is neither: it is a ladder of checks whose costs differ by orders of magnitude.

We propose a test-time search procedure for chart generation that treats verification as a cascade. The first step runs the code. The second step reads the rendered figure's own geometry to find overlapping text, clipped legends, and missing columns, which needs no neural model. Only the candidates that survive both steps reach a VLM critic. On top of this cascade sits a rule, adapted from active search, that grows each query's sampling budget in doubling rounds. Easy charts stop after one or two samples. Hard compositional charts get more. Hopeless queries abstain instead of burning the budget. The claim is that this cascade reaches the quality of fixed best-of-$k$ with a critic on every sample at a fraction of the cost. It should also beat that baseline on multi-channel and multi-panel requests at matched cost.

Core Claim

The closest prior work is Adaptive Generate-Rank-Verify. It frames test-time search as paying a cheap ranking cost per candidate and an expensive verification cost per check, and it grows candidate pools in doubling "shells". It reports being 2.9× cheaper on HMMT math and 5.5× cheaper on LiveCodeBench than the best fixed budget. Its verifier is one scalar unit test or answer check, and it notes that multi-stage verifier hierarchies are unexplored.

Visualization is a natural case of exactly that hierarchy. VisEval already splits chart checking into three stages: code validity, SVG-level legality, and GPT-4V readability. It shows that executable code often fails the later stages. VisEval uses these stages only to score models after the fact, never to steer generation.

The new element here is twofold. First, the middle tier is reference-free, so it can run at inference time without a gold specification. Second, the stopping and abstention rules use the pass/fail pattern across tiers, not a single score.

Our hypothesis is specific. A large share of the visual defects that VLM critics flag are also detectable from the renderer's bounding boxes and the plotted data. Examples include label collisions, legend clipping, wrong columns, and fractional ticks on count data. If so, the expensive critic only needs to see a small, pre-filtered set of candidates. If the hypothesis is false, the experiments will show it, and the cheap tier collapses into mere filtering.

Research Program

Generator. We use an open plotting-code model. The primary choice is Qwen2.5-14B, the base used by RL-Text2Vis. We will use PlotCraftor-30B from PlotCraft if its weights are available, which tests whether adaptive search still helps a model already fine-tuned for hard charts.

The three tiers. Each candidate program passes through tiers in order of cost.

  • Tier one executes the code in a sandbox.
  • Tier two inspects the rendered Matplotlib figure through the renderer's own objects. It computes pairwise overlaps of text bounding boxes, checks whether the legend and tick labels fall outside the canvas, compares plotted data against the columns and aggregations named in the query, and flags integer-valued data drawn with fractional ticks. A lightweight parse of the query supplies the expected columns, chart type, and sort order. This turns VisEval's gold-based legality check into a check that works at inference time.
  • Tier three is the critic from VIS-Shepherd. It is a Qwen-2.5-VL-7B fine-tuned on 2,700 expert critiques. It scored 3.41 on its 5-point critique evaluation, against 2.98 for the 72B base model, which makes it a cheap but credible final judge.

Expected cost per candidate. Let $c_1, c_2, c_3$ be the measured GPU-seconds of the three tiers, $c_g$ the generation cost, and $q_1, q_2$ the probabilities that a candidate passes tiers one and two. Then the expected cost per candidate is

$$ \mathbb{E}[\text{cost}] = c_g + c_1 + q_1 c_2 + q_1 q_2 c_3 . $$

The saving over critiquing every sample comes from the term $q_1 q_2 c_3$. That saving is large only if tier two rejects often, and correctly. We measure the $c$ values directly, including render latency, rather than treating each sample as a unit. Doing so addresses the cost-oversimplification gap noted for ADAP-style formulations.

Allocation rule. For each query, we generate candidates in doubling rounds of 1, 2, 4, 8, and so on, up to a cap. Within each round:

  • tiers one and two run on all new candidates;
  • the generator's log-likelihood ranks the tier-two survivors;
  • only the top survivors go to the critic.

The search stops at the first candidate the critic accepts. It abstains, returning the best tier-two survivor with a warning flag, if two consecutive rounds produce no tier-two survivor. The abstention rule targets the "impossible problem" trap, where adaptive search spends its whole ceiling on a query the model cannot solve.

Optional prior on starting budget. As a separable component, a small probe on middle-layer hidden states predicts the starting round. This follows Optimizing Reasoning Efficiency through Prompt Difficulty Prediction, which found layer 45 of 60 most predictive on math, and Learning How Hard to Think. We treat this probe as an ablation rather than as the core contribution. Whether hidden states anticipate visual failure before any code is written is an open empirical question, and the doubling schedule works without the probe.

Baselines

Do nothing. A single greedy sample with no verification. This is the floor, and it corresponds to how VisEval and RL-Text2Vis report zero-shot numbers. Published reference points are:

  • RL-Text2Vis reports a 14% final pass rate for the zero-shot starting point on Text2Vis.
  • RL-Text2Vis-14B reaches 29%, against 30% for GPT-4o.
  • VisEval reports GPT-4 at a quality score of 2.89/5 with roughly 75% pass rate.

We rerun the greedy baseline with our generator rather than borrowing these numbers.

Strongest uniform competitor. Fixed best-of-$k$, where every sample is executed and sent to the VIS-Shepherd critic and the critic's top-scored chart is returned. We sweep $k \in \{1,2,4,8,16\}$. This is the strongest competitor because it uses the same critic with no rationing, so any quality gap cannot be blamed on a weaker judge. Fairness comes from the same generator, prompts, temperature, and hardware, with cost measured in identical GPU-seconds including rendering. Beating it means dominating its cost-quality curve: equal final pass rate at clearly lower average cost, and higher pass rate on the hard subset at matched average cost.

Strongest existing adaptive method. ADAP applied as published, with the generator's likelihood as ranker and the VLM critic as its single verifier. No reported result found on Lacuna for ADAP, or for the marginal-reward allocator of Learning How Hard to Think, on any visualization benchmark. Their reported gains are on math and code only: 50% fewer samples at equal success, and the HMMT and LiveCodeBench savings above. Beating ADAP isolates the value of the cascade over a two-tier search.

Key ablations. Three variants isolate the main ingredients:

  • removing tier two, which yields two-tier ADAP;
  • removing tier three, so the search stops on the first tier-two survivor;
  • replacing the doubling schedule with a fixed budget behind the same cascade.

The cascade earns its place only if removing tier two raises cost substantially at equal quality, and removing tier three lowers quality measurably.

Why not just…?

Why not just use a bigger model? PlotCraft reports that scaling from 30B to 480B gives diminishing returns on hard compositional tasks. Its fine-tuned 30B model scores 6.16 on the PlotCraft benchmark against 5.49 for Qwen3-Coder-480B. Scale is therefore not a substitute for verification on the cases we target. We still include a larger-model, single-sample reference at matched GPU-seconds, so the comparison is made directly rather than asserted.

Why not just filter with execution and geometry checks and skip the VLM? This is the most serious threat, and the tier-three ablation tests it directly. If tier two catches nearly everything the critic catches, the honest conclusion is that cheap checks suffice and the VLM can be dropped. That would still be a useful, cheaper result. Two pieces of evidence point the other way:

  • RL-Text2Vis reports that removing its visual reward hurt chart correctness, even though code execution rose to 97% while the final pass rate stayed at 29%.
  • VIS-Shepherd documents defects such as unsuitable color scales and invisible points due to transparency, which bounding-box geometry cannot see.

We will report the fraction of independent-judge failures that only tier three detects.

Why not just set the budget from chart complexity, for example the number of encoded channels or panels? VisEval's finding that error rates jump at three or more visual channels makes this a sensible heuristic. We therefore implement it as a baseline: a budget lookup by parsed channel and panel count, run behind the same cascade. The doubling schedule should beat it when difficulty does not track surface complexity, such as short prompts with implicit aggregation or sorting, which VisEval reports cause failures. If the heuristic matches the doubling schedule, we will recommend it as the simpler choice.

Why not just run a fixed number of critique-and-repair turns? Fixed refinement rounds, as used in PlotCraft's multi-turn setting, spend turns on charts that were already fine. They can also oscillate on cosmetic edits. We run a fixed 3-turn VIS-Shepherd repair loop as a further matched-cost baseline. Combining it with the cascade's stopping rule is a natural extension, but we treat it as secondary.

Proposed Experiments

Benchmarks. Evaluation runs on Text2Vis, VisEval's 2,524 queries, and the 982 PlotCraft tasks. PlotCraft's Hard split and VisEval queries with three or more channels form the pre-registered hard subset.

Metrics. We report final pass rate (Text2Vis) and VisEval's validity, legality, and readability scores, alongside average and 95th-percentile GPU-seconds per query.

Judge independence. VLM judges carry their own perceptual biases. The judge used for final scoring must therefore never be the one that steered the search. Search uses VIS-Shepherd. Scoring uses VisEval's GPT-4V readability checker and PlotCraft's judge, which reports a Cohen's κ of 0.72–0.90 with human experts. A 300-chart human audit, stratified by easy, hard, and abstained queries, checks whether gains survive human eyes.

Decisive checks. Three results decide the outcome:

  1. Cost-quality curves for every method on the same axes.
  2. Matched-cost pass rate on the hard subset, the place where adaptive allocation should matter most.
  3. Tier-two precision and recall against the independent judge, which tells us directly whether cheap geometry stands in for perception.

We also measure abstention calibration: the share of abstained queries that no method at 4× the budget can solve. If abstentions are mostly solvable, the rule is cutting off solvable hard problems and must be revised.

Success criterion. The proposal succeeds if, at matched quality, average cost falls by at least half relative to fixed best-of-$k$, and if, at matched cost, the hard-subset pass rate rises.