Reading What Searchers Asked For: Co-Click Queries as a Perception Reward for Chart Reasoning

Multimodal language models, meaning models that take an image and text and answer in text, fail on charts in a specific way. They often answer hard trend questions correctly while misreading an axis minimum. They also lean on the chart's title. On Chart101, accuracy reaches 79% when the title contains the answer and falls to around 30% when it does not (Plot Twist). The strongest recent fix splits the system in two. A trainable "perceiver" writes a text description of the image, and a frozen text-only "reasoner" answers from that description. The perceiver is trained with reinforcement learning to write descriptions that help the reasoner get verified answers right (RAPID). That fix needs gold question-answer pairs, and gold pairs for real-world charts are scarce and expensive to collect.

Web search logs contain a signal about charts that no one has used for this purpose. When many different search queries lead users to the same page, the set of queries is a free, human-written list of what people wanted from that page. This idea is called co-click: two queries count as related because they send users to the same document. We propose to take pages that hold a single chart, gather the queries that users clicked through to reach them, and reward the perceiver when its chart description explains those queries better than the title and surrounding page text already do. The subtraction matters. Users click largely because of titles, so an unadjusted click reward would teach the model the title shortcut that already causes these failures. Only the part of the queries that the image itself can explain should earn reward.

Core Claim

Behavioral queries tell a perceiver what to read from a chart: which series, which years, which extremes. They cannot tell it whether it read those values correctly. Our claim is that this "what to read" signal, taken from co-click logs and measured as a residual beyond the title, improves elementary chart reading and shrinks the title-shortcut gap. It should also add to gold-answer rewards rather than duplicate them. The claim fails if queries generated by a language model from the same page text, with no clicks involved, work just as well. That comparison is the central experiment.

This differs from its closest neighbors on one specific axis each:

  • RAPID. It supplies our training setup (a perceiver trained with GRPO, a group-relative policy-gradient method, feeding a frozen reasoner), but its reward is correctness on the verified ViRL39K set. Ours is label-free.
  • Visual Search at Alibaba. It mines clicked and skipped items to train image matching, a task with no reasoning over image content.
  • PUGC. It turns user-generated posts into text preference pairs and reports that this breaks down on math and technical content. That motivates rewarding coverage rather than having a reward model judge the reasoning itself.
  • VeriSciQA. It generates question-answer pairs from the paragraphs that cite a figure and filters them with models. It uses document text as a proxy for figure meaning but no behavior. We treat it as the main competing hypothesis rather than as a component.

The Reward

For a chart $f$, let $t$ be its title, caption and surrounding page text, and let $Q_f$ be the distinct queries whose clicks led to its page. The perceiver writes a description $c$. A frozen text language model $p_\phi$ scores query likelihood, and the reward is

$$ r(c) = \frac{1}{|Q_f|}\sum_{q \in Q_f}\Big[\log p_\phi(q \mid c, t) - \log p_\phi(q \mid t)\Big] - \lambda\,\mathrm{len}(c), $$

where $\lambda$ sets a length penalty so the perceiver cannot win by listing everything.

The subtracted term is the title residual. It borrows the logic of click models from web search, which separate "the user clicked because of where the result was shown" from "the user clicked because the result was relevant." Those models are described in THUIR at NTCIR-13, where click-derived labels came within 0.0036 nDCG@10 of labels from human judges. In our setting, the title plays the role that display position plays in search.

A query that the title alone explains, such as the page's headline topic, contributes roughly zero reward. A query about a year, a sub-series or a peak value contributes reward only if the description surfaces that content. We also add a contrastive term, following the Alibaba practice of using skipped items as hard negatives. Each query should be more likely under its own chart's description than under descriptions of other charts from the same site.

Research Program

Data. The data comes from ORCAS: 18.8M clicked query-document pairs over 1.4M URLs, with click agreement against expert judgments of Kendall's τ = 0.622.

ORCAS has three limits we have to work around:

  • Positive edges only. It releases no click counts and no skipped results. We therefore replace the ≥10 co-click threshold of De Bona et al. with a minimum number of distinct queries per page.
  • No images. We recrawl the URLs and keep pages whose text still matches the corpus snapshot.
  • Clicks go to pages, not charts. To avoid guessing which chart a click was for, we start with pages that hold exactly one detected chart.

The first deliverable is a count of how many usable chart pages survive these filters. If fewer than about twenty thousand single-chart pages with at least three distinct queries survive, the project falls back to multi-chart pages. In that case each query is credited to the chart on the page whose description explains it best, a standard way of handling labels attached to a group rather than an item.

Model. The perceiver is Qwen2.5-VL-7B and the frozen reasoner is Qwen3-8B, matching RAPID so that numbers are directly comparable. The query scorer $p_\phi$ is a separate frozen text model, so the perceiver cannot tune the scorer to suit itself.

We train three variants:

  • the click reward alone;
  • RAPID's gold reward alone, as a reproduction;
  • a sum of the two.

Baselines

Naive reference points. The first is the untrained Qwen2.5-VL-7B answering directly. The second is the same model's untrained descriptions passed to the frozen reasoner. The gap between them shows how much the split into perceiver and reasoner helps before any training.

Strongest existing method. This is RAPID trained on ViRL39K. The file reports 53.2% average accuracy across its benchmark suite for Qwen2.5-VL-7B with a Qwen3-8B reasoner, an 11.2-point absolute gain (RAPID). We rerun it with identical GRPO steps, rollout counts and hyperparameter search budget as our method. The click-only variant is not expected to beat RAPID overall, because it never sees answers.

Beating RAPID means one of two outcomes:

  • the combined variant exceeds RAPID on the chart benchmarks at matched compute; or
  • the click-only variant matches RAPID's gains on elementary chart questions while using no gold labels.

Page-text queries without behavior. This ablation isolates the ingredient we claim is new. For each training chart, an LLM writes $|Q_f|$ queries from $t$, the page text. This follows VeriSciQA's approach of generating supervision from document text (VeriSciQA), which reports a +2.05% average gain across CharXiv, MMStar and MathVista. Everything else is identical, including the title residual. If behavior matters, the real queries should win on elementary reading.

Raw click reward. This drops the subtracted title term. It tests whether the residual is what stops the shortcut from being amplified.

Synthetic chart data. This is the LAMENDA synthetic-data approach, which raised MatCha on human-written ChartQA questions from 38.2% to 53.9% (Synthesize Step-by-Step). We use it as a supervised warm start for the perceiver under the same budget, to test whether clicks add value beyond template data.

Why not just…?

Why not just generate questions from the page text, as VeriSciQA does? This is the real threat. Page text and clicked queries may encode the same needs. Two considerations favor clicks. First, queries are written by people who have not seen the chart, so they ask about values rather than restating captions. Second, co-click sets gather many phrasings of one need. The page-text-query ablation above settles the question directly. If it matches our method within noise on Chart101 and CharXiv descriptive questions, we will report that clicks add nothing and recommend the cheaper pipeline.

Why not just remove titles from training images? The perceiver would still be rewarded for matching queries that were clicked because of the title. It would learn to guess title-like content from other cues, or hallucinate it. The residual removes only what the title explains. We will run title-masked training with the raw click reward as a further variant to check this.

Why not just use synthetic template data? Template-trained models score about 61% on machine-generated ChartQA questions but about 30% on human-written ones (ChartQA). CharXiv reports that synthetic benchmarks give an illusion of progress on real arXiv figures (CharXiv). Templates also need SVG or plotting code underneath, which raster web charts lack. The LAMENDA baseline tests whether clicks add value beyond templates.

Why not just prompt for step-by-step reasoning? On MATH-Vision, step-by-step prompting sometimes lowered accuracy (MATH-Vision). CharXiv shows that a misread first step turns the whole chain into hallucination. We include step-by-step prompting of the base model as a zero-cost reference.

Proposed Experiments

Evaluation sets.

  • Chart101. We score elementary and collective accuracy, reported separately for title-present and title-stripped charts. The file reports collective accuracy often below 25% and a gap of 79% versus around 30% between title-present and title-stripped cases. It contains no 7B perceiver-reasoner result on Chart101.
  • CharXiv. We score the descriptive and reasoning splits. Reported reasoning accuracy is 47.1% for GPT-4o and 29.2% for InternVL Chat V1.5.
  • ChartQA human-written questions. We score relaxed accuracy.
  • MathVista and MATH-Vision. These serve as out-of-domain checks that chart-focused training does not hurt math. The decoupled LLM-LMM system scores 52.6 on MathVision (Decoupling Reasoning and Perception).

Decisive checks. There are four:

  1. Title gap. The gap between title-present and title-stripped Chart101 accuracy should shrink under the residual reward and widen under the raw click reward. This is the direct test of the open question in the research file: does behavioral signal ground perception, or does it amplify linguistic shortcuts?
  2. Gain by question type. Gains should concentrate on question types that appear in the query logs, such as specific years, extremes and named series. Tick counting is rarely queried and should gain less. That dose-response pattern would link the gains to behavioral coverage rather than to generic extra training.
  3. Complementarity. The combined click-plus-gold variant should beat RAPID alone at matched compute. That would show clicks teach something gold correctness does not.
  4. Hallucination audit. We sample 300 descriptions and have annotators check every stated number against the chart. The concern is that a reward for coverage produces confident, invented values. If the rate of invented numbers rises relative to RAPID, the click reward must be paired with a correctness signal. We will report that outcome as a limit of behavioral supervision.