Checking the Visual Critic Against the Renderer

Language models that write plotting code, web layouts, or vector graphics are increasingly paired with a second model that looks at the rendered picture and says what is wrong. The second model is a vision-language model, meaning one that reads both images and text. The loop runs like this: write code, render it, critique the image, revise the code. The weak link is the critic. It is itself a model that hallucinates. It reports overlapping labels that do not overlap, missing data series that are present, and clipped axes that are fine. The code model then dutifully "fixes" things that were correct.

This proposal takes a simple observation seriously. When the artifact is a program, the rendering engine knows the true geometry and data of every drawn element. Matplotlib can report the pixel extent of every text label, and a headless browser can report the bounding box of every page element. We therefore require the critic to phrase its complaints as small typed claims about identified elements, such as "label 7 overlaps legend 2" or "series 3 is absent." Each claim is checked against the renderer's own record before it reaches the code model. Claims the renderer refutes are dropped. Claims it confirms are passed on together with the measured evidence. Claims it cannot check are passed on but marked as unverified. The main prediction concerns correct plots: unaudited loops damage a measurable fraction of plots that were already correct, and auditing should remove most of that damage without losing the fixes.

Core Claim

Current visual feedback loops treat the critic's natural-language output as ground truth. MatPlotAgent sends GPT-4V's free-text critique straight to the code agent. Volcano lets a single model critique and revise its own answer. Canvas-of-Thought renders a mutable page structure and critiques the snapshot, but never checks whether the critique is correct. Separately, The Critique of Critique found in text-only settings that model critics have high recall but low precision: they flag errors in parts of a solution that were correct. This finding motivates the proposal.

We claim three things. In code-to-image loops, a large share of critic complaints concern properties the renderer can decide exactly. Filtering critiques by rendered-state verification raises net improvement on MatPlotBench. Most of that gain comes from fewer regressions on already-good outputs, not from more fixes on bad ones. The idea differs from Multimodal Meta-Verifier, which trains a verifier to emit bounding boxes scored by overlap against annotated targets in text-to-image generation. Here no annotation or verifier training is needed, because the program's own render state is the reference. The critic proposes and the renderer decides.

Research Program

The system has four parts. The first is an overlay renderer. It executes the generated code, walks the figure object (Matplotlib artists, or document elements for SVG and HTML), assigns each visible element a short ID, and produces two images: the plain render, and a copy with faint ID tags drawn at each element. The design borrows from Visual Self-Refine, which draws a model's predicted coordinates back onto the chart so the model can inspect its own anchors. We reuse the device so the critic can name elements unambiguously rather than describe them vaguely.

The second part is a claim schema with about a dozen predicate types. Geometric predicates cover overlap, clipping, off-canvas placement, and minimum font size. Contrast predicates flag low contrast between elements. Data predicates cover a series being absent, a value mismatch against the data arrays in the code, wrong axis scale, and wrong chart type relative to the query. A free-text "other" type catches everything else. The critic, prompted with the query, both images, and the schema, emits a list of claims.

The third part is a deterministic checker that evaluates each typed claim against the render record. It uses text extents, artist data, axis limits, colormap values, and element bounding boxes. Each claim is labeled confirmed, refuted, or uncheckable.

The fourth part is a revision step. The code model receives only confirmed claims, with numeric evidence such as "overlap of 14 by 9 pixels," plus uncheckable claims marked as unverified. After revision, the same checks are re-run on the new render.

That re-check lets us measure something the reasoning-limitations literature predicts but no one has measured in this setting. Auditing Meta-Cognitive Hallucinations found that reasoning models, when an external intervention corrected the root-cause error, reversed their hallucinated conclusion in only 22.5% of cases. We call this reverting behavior "chain disloyalty." Here we can measure it directly as the fraction of confirmed defects the code model acknowledges yet leaves in place, or reintroduces in a later round.

We also analyze claim coverage: what share of critic claims, and of real defects, fall into checkable types. If most real defects are uncheckable, the method's reach is limited, and we will report that rather than hide it.

To summarize the headline quantity, let $F$ be the fraction of initially defective outputs that improve under the loop and $H$ the fraction of initially acceptable outputs that get worse (the harm rate). Net benefit is roughly $F \cdot p - H \cdot (1-p)$, where $p$ is the share of outputs that start out defective. We predict auditing mainly lowers $H$. The effect should therefore be largest for strong code models, where $1-p$ is large.

Baselines

The naive reference is single-pass code generation with interpreter self-debugging and no visual feedback. MatPlotAgent reports GPT-4 at 48.86 on MatPlotBench. The same paper reports 53.44 for its full pipeline without the visual feedback module, so both numbers anchor the "no critic" end.

The strongest existing method is the full MatPlotAgent loop with an unfiltered visual critic. It reports 61.16 for GPT-4 and 51.70 for Magicoder-S-DS-6.7B (up from 38.49), both from MatPlotAgent. It is the right competitor because it is the published render-critique-refine system on the exact benchmark, and our method changes only what happens between critic and coder. We will re-run it ourselves with identical code model, critic model, query-expansion step, and round limit, alongside quoting the published figure. We will use open-weight models (a Qwen2.5-VL critic, and Magicoder plus one stronger code model) and one proprietary configuration for comparison with the published numbers.

A second loop baseline follows Volcano's decide step. After each revision, the critic compares old and new renders and keeps the better one. This is the obvious cheap guard against harmful revisions.

A third baseline is critic self-consistency. We sample the critic $k$ times and keep only complaints that appear in a majority of samples. $k$ is chosen so its critic compute matches or exceeds our single critic call plus checking.

The key ablations follow. "Typed claims, no checking" passes all schema claims through, which isolates the effect of structured phrasing from the effect of verification. "Checker only, no critic" runs every predicate exhaustively as a linter, which isolates the critic's contribution. "Overlay removed" tests whether element IDs matter.

All conditions share queries, models, decoding settings, a maximum of three revision rounds, and the same prompt-tuning budget on a held-out development split. Beating the strongest baseline means three things: higher MatPlotBench score than the re-run MatPlotAgent loop, a lower harm rate, and no drop in fix rate. Each difference must be judged by paired bootstrap over the benchmark's 100 queries. Because 100 queries is small, the new injected-defect benchmark described below carries the precision claims.

Why not just…?

Why not just run a deterministic layout linter and drop the critic? This is the real threat. If most failures are overlaps and clipping, an exhaustive linter would do as well at lower cost. The "checker only" ablation settles it directly. We expect the linter to miss intent-level defects: wrong chart type, wrong data mapping, a missing requested panel. Only a model reading the query can raise these, although a data predicate can sometimes confirm them once raised. We also expect the linter to over-fire on overlaps that are harmless by design, such as annotations that intentionally sit on data. If the linter matches the audited critic on score and harm rate, we will report that the critic is unnecessary for this class of artifact. That would itself be a useful negative result.

Why not just sample the critic several times and take a vote? Voting removes idiosyncratic false alarms but not systematic ones. A critic that reliably misreads dense tick labels as overlapping will do so in every sample. Visual Self-Refine reports that models checking their own coordinates in text confidently asserted wrong values were right. That is a consistent error, not random noise. The compute-matched self-consistency baseline, broken down by defect type on the injected-defect set, will show whether voting fixes precision for systematic errors.

Why not just use a stronger critic model? Scale does not obviously solve perception. GPT-4o scored 1.71 average precision on the hard subset of ChartP-Bench while a 3B model with visual self-refinement scored 37.66 Visual Self-Refine. ViCrit reports diminishing gains from 7B to 72B on fine-grained critique. We will still run the loop with the strongest available critic, audited and unaudited. If the unaudited strong critic already has near-zero harm rate, the method's value shrinks to cheaper critics, and we will say so.

Why not just train the critic to be precise, as ViCrit or the Meta-Verifier do? Training helps and is complementary. But it needs synthetic error data or annotated regions, and it still produces a model that can be wrong. Rendered-state checking needs no training and is exact within its coverage. An audited trained critic is a natural follow-up. The confirmed and refuted labels our checker produces are also free supervision for that training.

Proposed Experiments

The first experiment builds an injected-defect set from the reference solutions in MatPlotBench, following ViCrit's recipe of planting single known errors. We edit correct reference code to introduce exactly one defect of a known type: occlusion, clipping, dropped series, wrong scale, swapped colors, or wrong chart type. We keep an equal number of untouched correct plots. We then measure critic precision and recall at the level of individual claims, in the style of The Critique of Critique's atomic information units, before and after auditing. The false-alarm rate on untouched plots is the number to watch.

The second experiment runs the full loop comparison on MatPlotBench. We report its automatic GPT-4V score, which MatPlotAgent found correlates with human scores at r = 0.876. Because that evaluator is itself a visual model and could share the critic's blind spots, we add blinded human ratings on a subset, plus the checker's own defect counts.

The third experiment plots score against revision round for each condition. We predict unaudited loops plateau or regress after the second round, while audited loops do not.

The fourth experiment tests transfer to vector graphics on the VCode benchmark, where the browser supplies element geometry. The only reported figure found is Canvas-of-Thought's +8.1% SigLIP improvement over standard chain-of-thought on the CV-Bench subset with GPT-5 Canvas-of-Thought. We will re-run its loop with and without auditing on an open backbone rather than compare to that proprietary number directly.

The decisive check is the harm rate on initially acceptable outputs. If auditing does not cut it substantially relative to the re-run MatPlotAgent loop, the central claim is wrong. If it cuts harm but the linter-only ablation performs as well, the claim reduces to "the renderer is a better critic than the model." If the chain-disloyalty measurement shows the code model often leaves confirmed defects in place, the bottleneck is on the revision side, and the proposal's next step shifts there.