Look Before Reading: Judging Where Chart Misreadings Become Wrong SQL

Picture an analytics assistant that is shown a dashboard chart and asked a follow-up question about the database behind it. It must read the chart, decide which tables, columns and values the chart refers to, write a query, and interpret the result. If it misreads one legend entry, it will often write a perfectly valid query over a real column that answers the wrong question. Nothing crashes, and the final number looks plausible. The automated judges used to audit such trajectories make this worse. A judge is a language model that grades another model's outputs. It reads the agent's trajectory in order, so it takes in the misreading as a premise and then approves every later step that is consistent with it. The judge has the same failure as the agent: a perception error early in the sequence propagates through the judge's own reasoning.

We propose a judging protocol and a benchmark built to measure and fix this. The protocol, extract-then-diff, works in two phases. In the first, the judge independently re-derives the grounded facts each step should rest on: values read from the image, schema elements looked up in the catalog, rows returned by execution. It does this without seeing what the agent claimed. In the second, it compares those facts to the agent's claims and uses the trajectory's dependency structure to label each faulty step as either the origin of an error or an inheritor of one. The central claim is that blindness and context serve different purposes. Blindness is needed for the perception check. Context is needed only for attribution. Mixing them, as current judges do, fails in a predictable way.

Core Claim and Why It Is Open

Two pieces of prior work disagree on how a verifier should treat upstream reasoning, and this disagreement motivates the proposal.

AgentEval represents agent runs as dependency graphs (directed acyclic graphs, where each step points to the steps whose outputs it consumed). It grades each step relative to its parents with a GPT-4o judge. It reports that 63% of step failures propagate from upstream, and that step-level judging raises failure-detection recall from 0.41 (end-to-end checking) to 0.89, with 72% root-cause accuracy against an 81% human ceiling. Its judge sees upstream context, and it covers only text and tool calls.

BeaconChart reaches the opposite conclusion for chart reading. Its verifier must not see the perception agent's reasoning; it re-extracts the key fields from the raw image. This raised the probability of catching an error from 27.95% to 54.15%. BeaconChart verifies only single-question chart answers. It does not attribute errors across steps, and it does not touch databases.

Perception-Judge names the underlying mechanism, response anchoring: fluent solutions built on a misperceived image fool multimodal judges. A base Qwen2.5-VL-7B detects such errors only 69.5% of the time, and GRPO training on ranked perturbations lifts this to 85.7%. That work, too, is single-turn.

Our hypothesis reconciles these results. A contextual judge in the AgentEval style will under-detect perception origins and blame the visible downstream symptom, usually the SQL. A blind judge in the BeaconChart style will detect the bad read but flag every downstream step as independently wrong, destroying root-cause attribution. Extract-then-diff should beat both on the quantity practitioners actually need, which is naming the step where the trajectory first went wrong.

The setting where this matters most is chart-to-database work. There, perception errors turn into what text-to-SQL research calls schema hallucination: queries that reference wrong or non-existent tables, columns or values. The text-to-SQL line treats the input as text only. This includes CRUSH4SQL, TA-SQL, QueryAgent and PRACTIQ. The chart line, including CharXiv and BeaconChart, never writes queries. CharXiv's finding that 20 of 24 models are near chance at counting labeled ticks suggests visual misreadings will be common upstream of the query.

The Key Distinction: Fabricated Versus Valid-but-Wrong Grounding

One kind of schema hallucination does not need a judge. If the agent invents a column that does not exist, a catalog lookup or an execution error catches it. QueryAgent's environmental feedback does exactly this, using empty results and exceptions as signals.

The hard case is a valid-but-wrong grounding. Suppose the agent misreads a legend color and filters on region = 'EMEA' when the chart showed 'APAC'. Both values exist in the database, so the query executes. Only the image can reveal the error. We therefore split every analysis by error type. We expect deterministic checks to match any judge on fabricated identifiers, and we claim an advantage only on valid-but-wrong errors that originate in perception. If the protocol's gains appear only on fabricated identifiers, the proposal has failed.

Research Program

Benchmark: ChartSQL-Trace. We build charts from known SQL over BIRD databases, the cross-domain benchmark used by TA-SQL. A gold query produces a result. The result is rendered as a chart whose difficulty features follow CharXiv's documented weak spots: dual axes, dense legends, multiple subplots, unlabeled ticks. A templated follow-up question then requires reading the chart before querying further, for example "for the category with the steepest decline, list its top five accounts." Because the chart was generated from SQL, every intermediate fact has a gold value: which series exists, which value it encodes, and which column and literal it maps to.

Trajectories follow a fixed step grammar: read chart, link to schema, write SQL, execute, interpret. This makes two labels computable deterministically for every step. The first is absolute correctness. The second is conditional correctness, meaning whether the step is right given its parents' actual outputs. A step that is absolutely wrong but conditionally correct is an inheritor. A step that is conditionally wrong is an origin. This replaces most of the human annotation AgentEval needed. We will still have two annotators label a 300-trajectory sample to check that the grammar captures real failures.

Sources of errors. Trajectories come from two sources. The first is natural runs of several open and proprietary multimodal models. The second is controlled injection: at a chosen step $k$ we replace the agent's output with a perturbation and let the agent continue from it. Perturbations come in two kinds. Valid-but-wrong perturbations swap in a different existing series, value or column. Fabricated perturbations insert a non-existent identifier. Injection lets us draw the curve we care most about: how often a judge accepts a step as a function of its distance from the origin error.

Protocol. Extract-then-diff has two phases.

  1. Blind grounding. For each step, the judge receives only that step's raw inputs: the image, the catalog slice, and any execution output. It writes down the facts the step should produce.
  2. Diff and attribution. A second call compares those facts with the agent's claims. It then walks the dependency graph and labels each mismatched step as an origin if its parents' claims were correct, and as an inheritor otherwise.

Catalog lookup and execution are given to the judge as tools, so the fabricated case is handled cheaply and the model's effort goes to perception. A secondary text-only slice tests whether the protocol transfers when the "perception" is reading an ambiguous utterance rather than an image. That slice uses PRACTIQ dialogues and BIRD-INTERACT tasks, where Claude 3.5 Sonnet detects ambiguity at 77.4% against roughly 94% for humans.

Downstream use. We also use the judge at run time. When it flags an origin step, the agent re-performs that step only and continues from there. We measure final execution accuracy.

Baselines

The naive reference is end-to-end checking: the trajectory passes if its final result set matches gold. AgentEval's 0.41 recall on its own production workflows indicates how weak this is. On ChartSQL-Trace it can detect failure but cannot localize it, so its root-cause accuracy is defined as always blaming the final step.

The deterministic baseline is a catalog-and-execution checker in the QueryAgent style. It flags non-existent identifiers, execution exceptions and empty results. This is the strongest competitor on fabricated errors, and we expect it to tie with or beat every judge there. On valid-but-wrong errors it should be near zero by construction. Showing this empirically is part of the result.

The strongest existing judge method is AgentEval-style contextual step judging. A frontier multimodal model with access to the image receives the full trajectory up to each step and a step-specific rubric, and grades steps relative to their parents. AgentEval reports 0.89 failure-detection recall and 72% root-cause accuracy on its internal workflows. There is no reported result on Lacuna for this method on any multimodal or text-to-SQL trajectory set, so we will run it ourselves.

The blind-verifier baseline applies BeaconChart's independent re-extraction at every step and marks any mismatch as a failure, with no attribution phase. BeaconChart reports error-identification probability rising from 27.95% to 54.15% on its chart task.

The trained-judge baseline fine-tunes Qwen2.5-VL-7B as a contextual trajectory judge with Perception-Judge's GRPO ranking reward. The reward requires that a correct trajectory rank above one with a perception perturbation, and that one rank above a trajectory with perception plus reasoning corruption. Training uses our injected trajectories from a disjoint set of databases.

For fairness, all prompted judges use the same backbone models and receive the same tool access. They get the same total token budget per trajectory, with extract-then-diff's two calls counted together. Rubric and prompt tuning for every method uses the same 200-trajectory development split. The trained judge is compared with extract-then-diff run on the same 7B backbone.

Two ablations isolate the key ingredient. The first removes blindness by letting the phase-one extractor see the agent's claim. The second removes attribution, which recovers the blind verifier.

Beating the field means three things on valid-but-wrong, perception-origin errors. Extract-then-diff must exceed the contextual judge's root-cause localization accuracy by at least 10 points. It must exceed the blind verifier's origin-versus-inheritor accuracy by a similar margin. And it must not fall below the deterministic checker on fabricated errors.

Why not just…?

Why not just check the final query against the database? Execution catches fabricated identifiers and crashes. It cannot see that a valid filter value is the wrong one, and it cannot say which step went wrong. AgentEval's 0.41 end-to-end recall reflects the first limitation. Our split by error type will measure the second directly. We expect execution checks to win on fabricated errors and to be blind to valid-but-wrong ones, and we will report both halves.

Why not just prompt a frontier judge to think step by step over the whole trajectory? This is the contextual baseline, and it is a real competitor because frontier models may resist anchoring better than the 7B models studied by Perception-Judge. The evidence against it is indirect. Perception-Judge documents anchoring on fluent solutions. BeaconChart's verifier improved only when it was blinded. VCBench reports that reasoning chains over misperceived images rationalize the error. The acceptance-versus-distance curve settles the question. If the frontier contextual judge's false-acceptance rate does not rise for steps after an injected misreading, anchoring is not the problem and our protocol's motivation collapses.

Why not just train a judge on perturbed trajectories, as Perception-Judge did? Reward-trained judges may learn to discount the trajectory on their own, which would make the explicit blind phase redundant. We take this threat seriously, which is why the trained judge is a baseline at matched backbone size. If it matches extract-then-diff on localization, the honest conclusion is that training substitutes for protocol. In that case we would report the protocol as the zero-training option rather than the better one.

Why not just blind the verifier everywhere, as BeaconChart does? Blind checking finds bad reads, but every step downstream of one also mismatches gold. A blind verifier therefore reports a cascade of failures with no origin. The attribution ablation measures exactly how much localization this costs.

Decisive Checks

Three outcomes would decide the proposal.

First, the anchoring curve. For contextual judges, the probability of accepting a step should rise with its distance after an injected valid-but-wrong misreading. Extract-then-diff should stay flat. If the contextual curve is also flat, the premise is false.

Second, the error-type split. Extract-then-diff should gain on valid-but-wrong perception origins and merely tie with the deterministic checker on fabricated identifiers. A gain confined to fabricated errors would show the judge is redundant with a catalog lookup.

Third, run-time repair. Re-running only the step the judge names as the origin should raise final execution accuracy on ChartSQL-Trace more than re-running the step the contextual judge blames, at equal call budget. This would show that better localization matters for fixing agents, not only for grading them. The PRACTIQ and BIRD-INTERACT slice then tests whether the same separation between blind checking and contextual attribution helps when the misread input is an ambiguous request rather than a chart.