Where Did the Chart Reader Go Wrong? Learning from Mixed Human Feedback on Perceptual Drafts

When a vision-language model answers a question about a chart wrongly, the error usually starts before any reasoning happens. The model misread a tick label, confused two colored series, or pulled a number from the wrong subplot. Its later reasoning then treats that misreading as a fact. Today's fixes collect feedback on the final answer, such as "this answer is wrong" or "answer A is better than answer B." That kind of feedback cannot tell the model which step failed. We propose a different setup. The model first writes out a perceptual draft: a short list of the values it read, where in the image it read each one, and which visual element each belongs to. Human editors then give cheap, varied feedback on that draft. They can flag a wrong item, click where the right value is, fix a single number, or pick the better of two drafts. All of these signals are combined in one probabilistic model that learns which perceived items are wrong. The result is a learned "perception critic." It is used to fine-tune the reader and, at inference time, to re-read or discard doubtful evidence before reasoning starts.

This matters because the perception step is now the main thing holding back chart and visual-math reasoning. Models that fail descriptive questions on real arXiv figures almost never solve the reasoning questions on them (CharXiv). About 62% of Gemini 2.0 Flash's errors on elementary visual math are perception errors (VCBench). Human editing is known to be far cheaper than annotating from scratch, but it has mostly been used for masks and boxes. It has not been used to repair the perceptual inputs of a reasoning model.

Core Claim

Feedback on the final answer is only partially identifiable with respect to where the fault lies. Feedback that points at specific items in a perceptual draft removes that ambiguity. So, for the same amount of human time, the second kind of feedback should train a better chart reader.

The argument borrows from reward learning. Skalse et al. show that a single feedback type leaves many reward functions equally consistent with the data, and that combining incomparable feedback types shrinks this set. Chart QA has the same structure. Write $c_\theta(e_i \mid x)$ for the critic's probability that perceived item $e_i$ of image $x$ is correct, and $U$ for the set of items the reasoning step uses. A final-answer label then constrains only a product:

$$p(\text{answer correct}) = \epsilon_r + (1-\epsilon_r)\prod_{i\in U} c_\theta(e_i \mid x),$$

where $\epsilon_r$ is the chance that reasoning reaches the right answer despite bad evidence. Many assignments of blame to individual items fit the same answer label, and blaming the reasoning step fits it too. A flag on item $i$, a click, or a cell edit constrains $c_\theta(e_i\mid x)$ directly.

We fit all signals jointly by maximizing

$$\mathcal{L}(\theta)=\sum_{k}\sum_{m\in M_k}\log p_m\big(f_{k,m}\mid c_\theta(\cdot\mid x_k)\big).$$

Here $k$ indexes examples, $M_k$ is the set of feedback types collected on example $k$, $f_{k,m}$ is the observed feedback of type $m$, and each $p_m$ is a likelihood written for that type:

  • Flags and cell edits are Bernoulli evidence that the item is wrong.
  • Clicks are a location likelihood around the true region.
  • Choices between two drafts follow a Bradley–Terry model on summed log-critic scores.
  • Answer labels follow the equation above.

This follows the design of MAVRL, which gives each feedback type its own likelihood so that their relative weights come from their informativeness rather than hand tuning. MAVRL was tested only on control tasks with simulated humans. We move the idea to multimodal perception and use real annotators.

Research Program

We build three things.

The perceptual draft format. The draft is not a bare table. ChartQA shows that table-only pipelines fail on questions that refer to visual attributes like "the orange line." ChartVLM shows that a cascaded table extractor's mistakes cannot be recovered downstream. Each draft item is therefore a value, a bounding region, and an attribute binding (series, color, subplot). The base model, a 7B-class open vision-language model such as Qwen2.5-VL-7B, is prompted and lightly fine-tuned to emit this format before answering.

The annotation loop. A routing rule decides which drafts reach humans. It uses critic uncertainty plus disagreement from an independent re-extraction step in the style of BeaconChart, whose verifier re-reads the image without seeing the first draft. For each routed draft, an annotator chooses the cheapest feedback that resolves it. Annotator time is logged per action, so that every comparison can be made at equal human cost. INQUIRE showed that the best query type shifts from expensive to cheap as an agent's uncertainty falls, again with simulated users. We measure whether that pattern holds for real people editing chart drafts.

Human editors anchor on model drafts. To measure this, a subset of items is flagged "blind": the annotator sees only the question and the highlighted region, not the model's value.

The critic in use. The fitted critic is used in two ways. During training, it serves as a dense, item-level reward for reinforcement fine-tuning of the drafting step. At inference, items the critic scores below a threshold are re-read from a cropped region, a small concrete version of the active visual querying argued for in Position: Reasoning After Perception Means Reasoning Without Vision. If re-reading does not raise the score, the item is dropped. Items from subplots not referenced by the question are excluded before reasoning.

That last use is where contextual noise enters. In text retrieval, removing irrelevant context helps a great deal: Superposition Prompting prunes uninformative document paths, and DAS down-weights answers drawn from low-relevance documents. CharXiv reports a 30–50% accuracy drop for open models on figures with six or more subplots. We test whether critic-guided pruning of visual evidence reduces that drop.

Data. Training uses ChartX images, whose gold CSV tables let us simulate feedback exactly. It also uses real scientific figures from VeriSciQA for the human study. CharXiv, ChartQA, the ChartX test split, and VCBench stay held out. Simulated feedback lets us sweep budgets cheaply. The human study then measures how far real feedback departs from simulation, a gap MAVRL and INQUIRE left open.

Baselines

Base model. The naive reference is the base model answering directly, with and without step-by-step prompting. VCBench reports that step-by-step prompting can hurt on perception-heavy problems, so both versions are needed.

BeaconChart. The strongest existing method is BeaconChart. Its planner, perception agent, and independent verifier turn mismatches into stored corrective instructions. With Qwen3-VL-32B it reports 72.05% accuracy on ChartX and 91.60% on ChartQA. That ChartX result is 8.6 points above the plain backbone, which implies roughly 63.45%. Its verifier's rate of flagging an error when the answer failed rises from 27.95% to 54.15%. BeaconChart is the right competitor because it attacks the same problem, cascading perception errors, with no human cost at all. If automated verification is enough, our human loop is wasted effort.

We rerun BeaconChart on our 7B backbone with the same inference-call budget per question. No reported result found on Lacuna for BeaconChart on a 7B backbone or on CharXiv. If compute allows, we also run both methods on Qwen3-VL-32B to anchor against the published 72.05%. Beating BeaconChart means higher accuracy on the CharXiv reasoning split and on ChartX at matched backbone and inference budget. For reference on CharXiv, the paper reports 47.1% for GPT-4o and 29.2% for the best open model.

Answer-level preference tuning. This baseline spends the same logged human minutes on final-answer judgments and pairwise answer preferences, then trains with direct preference optimization. A second variant uses an outcome-reward recipe like Logos from MIRAGE, which lifted Qwen2.5-VL-7B from 28.8% to 37.1% on MIRAGE. Answer feedback is fast to give, so at equal time this baseline gets more labels.

Supervised table fine-tuning. This baseline uses annotators' time to transcribe full gold drafts from scratch, followed by ordinary supervised training.

Ablations. Three ablations isolate the key ingredients:

  • Replacing the joint likelihood with a hand-weighted sum of per-type losses, tuned with the same budget.
  • Restricting feedback to a single type (flags only, or edits only).
  • Turning off inference-time re-reading and pruning, which separates the training benefit from the noise-filtering benefit.

All methods share the backbone, the training images, the number of gradient steps, and a fixed hyperparameter search budget.

Why not just…?

Why not just fine-tune on gold tables? On ChartX this is close to an upper bound, since the gold tables come free with the dataset. That is why ChartX serves only as the simulation testbed. On real scientific figures no gold tables exist, so someone must produce them, and editing literature such as the Dresden OCT dataset reports more than a 15x time saving from correcting drafts instead of drawing from scratch. This is a real threat, so the human study compares transcription against editing at equal annotator minutes. If transcription matches our method there, the varied feedback types are unnecessary, and we will say so.

Why not just use an automated verifier like BeaconChart? Its own numbers show the verifier catches only about 54% of failures. Its authors also note that when the planner targets the wrong visual fields, the verifier checks the wrong evidence. Our ablation feeds BeaconChart-style verifier disagreements into the same critic as automatic flags, with no humans. The gap between that variant and the human-fed critic measures what people add beyond self-verification. If the gap is negligible, the human loop is not worth its cost.

Why not just collect answer-level preferences? They are cheaper per label, and this is the most plausible threat. The identifiability argument predicts that they learn slowly when perception and reasoning both fail. It does not predict that they lose at every budget. Human-time-matched learning curves will show where, if anywhere, item-level feedback pulls ahead.

Why not just add loss terms with fixed weights? MAVRL found that fixed weights break when one feedback channel is noisier than expected. Real annotators' click and edit noise is unknown in advance. The weighted-sum ablation tests this directly.

Proposed Experiments

Learning curves at equal human time. The first decisive check compares accuracy on the CharXiv reasoning split and on ChartX against logged annotator hours, for item-level mixed feedback versus answer-level preferences versus transcription. The core claim predicts that the mixed-feedback curve dominates at low budgets.

Fault localization. The second check tests whether the critic finds the faulty item. On a held-out set with known gold values, we measure how often the lowest-scored item is the wrong one. On MIRAGE questions, where perception was correct and the reasoning failed, we check whether the critic correctly avoids blaming perception. An answer-only critic should show the ambiguity the equation predicts: it should spread blame across items or misassign it.

Contextual noise. The third check is the slope of accuracy across CharXiv subplot counts, with and without critic-guided re-reading and pruning. We also measure transfer to multi-image VCBench, where Qwen2.5-VL-72B scores 48.08% and humans score 93.3%.

Realism of the feedback. The fourth check compares critics trained on simulated and real feedback with the same item counts. It also compares blind and draft-shown flags, which tells us how much anchoring bias reaches the learned critic.

What would refute the proposal. The claim fails if any of these hold:

  • Answer-level preferences match item-level feedback at equal human time.
  • Verifier-only flags close most of the gap to human-fed flags.
  • Real-human feedback degrades the critic below the BeaconChart baseline.