When a vision-language model solves a chart or diagram problem step by step, it increasingly attaches citations to its steps, such as "the peak is 42 [crop 3]." The natural interface for checking such a chain is a hover preview: point at the citation and see the cited image region. This proposal starts from an uncomfortable prediction. Models often produce a valid-looking citation next to a value they made up, and a preview that only shows the cited source is then likely to make the fabricated claim look more trustworthy. We propose hover cards that show the claim beside what the evidence actually says. Each card displays the stated value or relation next to the value an automated checker extracted from the cited crop, and it carries the checker's verdict. Visual cues placed before any hover steer the auditor toward the step types that automated checks cannot verify, above all spatial and relational claims. The auditor can then repair the chain with a small set of fixed operations (drop this evidence, re-run this step, abstain) instead of free-text feedback. The chain is re-executed after each repair.
The opportunity sits in a specific seam. Automated provenance checking of multimodal agents now works well for numbers and named entities. It rests on deterministic string matching, though, and gives no help for spatial reasoning errors. Those errors do not shrink with model scale. Separately, correcting only the faulty intermediate steps by hand recovers most failed answers. What is missing is a low-effort way for a person to find those steps. The main result we would test is whether a verification-aware hover lets a human catch the errors that the checker cannot, at a time cost comparable to reading the chain, while avoiding the trust inflation that plain citation previews cause.
Grounding and Prior Work
Two bodies of work motivate the proposal. LEDGERMIND supplies the method we build on. It records every tool output (text read from the image, crop coordinates, search snippets) in a structured evidence ledger. It forces reasoning steps to cite active ledger entries and checks them with an entity-consistency check and a numeric-coherence check. It names "phantom grounding," a valid citation attached to a fabricated value, and "repair-time amplification," where free-text self-correction adds new hallucinations. It also shows that typed repair operators avoid the second problem. Its ledger has no human-facing surface, and its checks rely on token matching and alias tables, which cannot verify paraphrased or spatial claims. MIRAGE supplies the two facts that make a human worth adding. First, manually correcting only the reasoning chains of failed problems restores the right answer about 70% of the time. Second, scaling reduces factual retrieval errors but has negligible effect on spatial reasoning errors such as left versus right. More Thinking, Less Seeing? adds a mechanism for why long chains need step-level visual checking: as a chain grows, the model's attention drifts from image tokens toward its own generated text. Plot Twist shows that models fail elementary coordinate reading while appearing competent on harder questions, because they lean on chart titles. That means the numbers inside a chain are exactly where silent errors hide.
On the interface side, work on citation and document previews both motivates and constrains the design. Piccardi et al. found that Wikipedia readers engage with citations more when hover previews are offered. Envisioning the Next-Gen Document Reader embeds entity links and citation warnings directly on text spans, and itself flags wrong entity links as a risk it does not address. Fuse shows in a 22-month deployment that compact cards with hover "Peek" are adopted for inspecting nested evidence. All three assume that the thing being previewed is what it claims to be. Phantom grounding breaks that assumption, which is why a preview must show a comparison and not just the source.
Two further papers constrain the design. Dumais et al. observed that triggering previews can cost more effort than scanning persistent text. That forces us to test an inline condition honestly instead of assuming hover wins. Aliannejadi et al. show that small visual markers help users anticipate where useful previews are. We reuse that idea for triage cues that tell the auditor which steps need a look before hovering.
Dataset transparency enters in two roles. The first is as the source of ground truth. We need chains whose step-level truth is known independently of any model judge. VeriSciQA (datajuicer/VeriSciQA) anchors each scientific figure question to author-written text and discarded 97% of candidates to keep only verified pairs. Chart101's synthetic plots give exact coordinates. The second role is as an output. The audited-chain corpus will be released with per-step provenance labels and an explicit record of which errors were injected and which occurred naturally. It will also document every preprocessing and error-injection parameter. We follow the lesson of Fell et al., whose reproductions collapsed (slide accuracy 92% to 54.7%) because preprocessing went undocumented.
Core Claim
There are two claims, and the second depends on the first. First, a hover preview that renders only the cited evidence raises human acceptance of phantom-grounded steps compared with no preview at all. Second, a hover card that shows the claimed value or relation next to the extracted evidence value, carries the automated verdict, and is paired with triage cues lowers false acceptance below both the no-preview and plain-preview conditions. The gain should be concentrated on spatial and relational steps, where rule-based checks abstain. The resulting human-plus-checker system should reach higher final-answer accuracy than the checker alone, at a reviewing time within a small factor of simply reading the chain.
Research Program
The backend is a LedgerMind-style pipeline run on open models, Qwen2.5-VL-7B and InternVL (OpenGVLab/InternVL). Every step is tied to ledger entries, and the entity and numeric checks are reimplemented as described. We add one component: a step classifier that tags each step as numeric, entity, spatial/relational, or deductive. A step's tag determines whether a rule-based check applies. Steps the checker has verified are rendered quietly. Steps the checker flagged, or cannot check at all, get a pre-hover marker.
Hovering a step opens a card with four elements. The first is the cited crop, drawn as a box on the full image instead of cut out, following DCoT's finding that boxes preserve context better than cropping. The second is the claim's value or relation. The third is the value or relation independently extracted from that crop by a separate extraction call. The fourth is the checker's verdict. For spatial claims, the card overlays the two referenced objects with an arrow for the stated relation, so a left/right swap is visible at a glance. From the card, the auditor can apply DROP, RETRY with the step's original inputs, or ABSTAIN. The chain then re-executes from that point under the ledger constraints, so a human repair cannot introduce uncited facts.
Test chains come from MIRAGE, Chart101, and VeriSciQA. Each is a mixture of naturally failed chains, labeled per step by two annotators with the dataset's ground truth, and correct chains with one injected error. Injected errors come in two kinds. Phantom-grounding errors keep the citation and alter a cited number or entity. Spatial errors swap a relation. Injection gives a known denominator for detection rates. Natural failures keep the study honest about real error distributions.
Baselines
The naive reference point is the chain shown as plain text with citation markers and no preview. Auditors read it and mark steps they believe are wrong. Every other condition must beat this on detection rate, or the interface has no reason to exist.
The strongest automated competitor is the LedgerMind protocol alone: entity and numeric checks plus automated DROP/RETRY/ABSTAIN, with no human. It is the strongest because it already addresses phantom grounding on numbers and entities at zero human cost. The file reports gains of +11.2 to +19.7 points on Hard-200 across backbones, and 58.9% on VTC-Bench with Gemini-3-Flash (LEDGERMIND). We run it on exactly our chains, with the same backbone and the same retry budget as the human condition. For accuracy on MIRAGE, the reference numbers are the base Qwen2.5-VL-7B at 28.8% and Logos-7B at 37.1% (MIRAGE). The human oracle ceiling is roughly 70% recovery on failed problems. Beating the checker means higher final-answer accuracy on the same chains, and the gain must be concentrated on spatial/relational failures.
The strongest interface competitor is a plain citation hover that renders the cited crop, in the style of Wikipedia reference previews (Piccardi et al.). It has the same latency, the same crop rendering, and the same repair operators, but no side-by-side comparison, verdict, or triage cues. The file contains no reported result on Lacuna for the effect of citation previews on acceptance of fabricated claims, so this condition also tests the harm hypothesis directly. A persistent inline condition shows every card's content expanded under each step, which tests the Dumais objection.
An LLM-judge condition gives a strong model the same card contents and asks it to flag steps. This tests whether a person is needed at all. Two ablations isolate the key ingredient. One removes the side-by-side extracted value but keeps the verdict and cues. The other removes triage cues but keeps the full card. A final repair ablation replaces typed operators with free-text feedback to the model, which tests repair-time amplification under human direction.
All human conditions use the same chains, counterbalanced within subjects, with the same per-chain time limit. All automated conditions share the backbone and compute. Primary metrics are false acceptance of erroneous steps, detection of erroneous steps, time per chain, and post-repair final-answer accuracy. All are broken down by step type.
Why not just…?
Why not just run the automated checks and let the system abstain? This is the real threat. If LedgerMind's checks catch nearly all consequential errors, the human adds cost and nothing else. The evidence against it is structural but not yet conclusive. The checks are deterministic token and alias matches that cannot evaluate spatial relations, and MIRAGE shows spatial errors persist regardless of scale. The per-step-type breakdown settles it. If checker-alone accuracy on spatial/relational failures matches the human-plus-checker condition, we would drop the triage-to-human design and report that the checker suffices.
Why not just show a plain citation preview, which users already like? Engagement is not accuracy. Under phantom grounding, the cited crop is real, so a plain preview gives visual confirmation to a false number. The plain-hover condition measures this directly. If it does not raise false acceptance over no preview, the first claim is wrong, and the contribution shrinks to efficiency.
Why not just display all evidence inline? Dumais et al. show that hover can cost more than scanning. On short chains, inline display may win, and we expect to report that. Fuse's deployment and the working-memory findings of Schneider et al. suggest that inline display degrades as chains lengthen. We therefore stratify results by chain length to locate the crossover instead of assuming it.
Why not just train a better-grounded model, or use an LLM judge? Training approaches such as Logos raise MIRAGE accuracy only to 37.1% and give no per-step audit trail. MIRAGE's own judge-based score inherits the judge's blind spots. The LLM-judge condition gets identical card contents. If it matches human auditors on spatial steps, the human is unnecessary, and the card becomes an input format for automated verification. That would still be a useful result.
Decisive Checks
The proposal succeeds or fails on three comparisons. First, plain hover must show higher false acceptance of injected phantom-grounding steps than no preview. Second, the full card must show lower false acceptance than plain hover, inline display, and both ablations, with the largest margin on spatial errors. Third, human-plus-checker must beat checker-alone and the LLM judge in post-repair accuracy on MIRAGE and Chart101 natural failures, at a median review time no more than about twice that of plain reading. Typed repair should also show fewer newly introduced unsupported steps than free-text human feedback, measured against the ledger. If the second comparison holds only for numeric errors the checker already catches, the human-audit premise fails, and we would report that outcome as such.
References
- LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger
- MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Quantifying Engagement with Citations on Wikipedia. (Part 2) (The translation and original text of the article are presented)
- Envisioning the Next-Gen Document Reader
- Fuse: In-Situ Sensemaking Support in the Browser
- Classified displays of web search results.
- Say it with Emojis: Co-designing Relevance Cues for Searching in the Classroom
- VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- Reproducibility of deep learning in digital pathology whole slide image analysis
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- fulgeo-Design of an Intuitive User Interface for a Multimedia Search Engine