When a model is unsure, a common trick is to ask it the same question several times and trust the answer only if the samples agree. Human annotation research has shown why this can fail. Annotators who share a blind spot agree with each other and are still wrong: crowd workers converge on incorrect labels when they share a heuristic or lack the needed context. We argue the same failure happens inside multimodal question-answering systems, and that it becomes acute in multi-turn, multi-hop settings. Suppose a smart-glasses assistant misreads a blurry storefront in turn one. Every later reasoning chain, in every sample, starts from that same misread entity. The samples agree, and the system answers confidently and wrongly when it should have abstained.
The proposal is to make agreement count only when it comes from evidence paths that do not share an upstream step. The system keeps an explicit graph of its reasoning across turns: what it perceived, what it retrieved, and which sub-answers each later step depends on. An answer earns confidence only from chains that reach it through different perception and retrieval nodes, for example a visual-entity lookup on one side and an OCR-then-web-search route on the other. When independent chains contradict a node inherited from an earlier turn, the system revises that node instead of propagating it. We test this on benchmarks whose scoring rewards abstaining over hallucinating, so the value of knowing when one is wrong can be measured directly.
Core Claim
In multi-hop multimodal dialogue, raw agreement among sampled answers overstates reliability, because most disagreement-free failures trace back to a single shared upstream error, usually in perception. This is the consensus-accuracy gap from annotation studies, reproduced inside a model. A conversation-level reasoning graph makes the shared causes visible. Measuring agreement only across node-disjoint paths in that graph should yield better abstention decisions and fewer multi-turn error cascades than vote counting, verbalized confidence, or uncertainty from token probabilities, at the same number of model calls.
To make this precise, let $K$ be the number of reasoning chains the system runs for the current question, and let each chain be a path from the question to an answer through perception, retrieval, and sub-answer nodes in the conversation graph. For a candidate answer $a$, define
$$s(a) = \frac{1}{K}\,\max\{\,|S| : S \text{ is a set of chains ending in } a \text{ that share no node below the question}\,\}.$$
The system answers with the $a$ that maximizes $s(a)$ when $s(a) \ge \tau$, and abstains otherwise. The threshold $\tau$ is tuned on a development split. Ordinary vote share is the special case in which shared nodes are ignored.
Grounding in Prior Work
The motivating evidence comes from annotation research. Crowd workers split search queries into equal halves rather than following linguistic structure, which yields high agreement on wrong segmentations (Crowd Prefers the Middle Path). Lay annotators share scientific misconceptions on biomedical extraction (Predicting Annotation Difficulty). Annotators shown emotional utterances out of order agree on surface acoustic cues (Muse-ing on Utterance Ordering). The common thread is that agreement fails when the agreeing parties share an input or a shortcut. Our contribution is to locate the shared input explicitly, as a node in a reasoning graph, rather than inferring it statistically.
Voting or Consensus? both motivates and constrains the design. It compares how multiple language-model agents settle on answers and finds that forcing consensus hurts reasoning badly: on MuSR, consensus reaches 28.4% against 54.8% for voting, while consensus helps modestly on factual recall. The paper attributes this to drift and sycophancy across debate rounds. We take two lessons from it. Chains should be generated without talking to each other, and aggregation should not require unanimity. What that paper does not consider is that independently sampled chains can still be correlated through shared evidence, and that correlation is exactly what we target.
The method borrows its memory structure from CMT-RAG. That system stores a multi-turn conversation as a persistent directed acyclic graph of sub-questions, search keywords, and cross-turn dependencies, and reports 41.73 exact match on the long-dialogue split of MuMu-QA. Its authors note that one wrong early decomposition breaks the whole chain. CMT-RAG is text-only and uses the graph for retrieval and speed, not for confidence. We extend the graph with perception nodes, which take image regions, OCR output, or recognized entities, and we use the graph's topology to decide when the evidence is genuinely redundant. Earlier graph-based dialogue models build edges from speakers or from coreference, such as DialogueGCN and the history graph in GoG. Those graphs feed a neural encoder. Ours is an explicit provenance record that we query, not one we embed.
The cross-modal gap is the reason a single node can poison many chains. CRAG-MM shows that retrieval recall drops from 58% on curated images to 51.9% on egocentric captures, and that 27% of multi-turn sessions end early after two consecutive failed turns. Work on visual-semantic embeddings (Learning Robust Visual-Semantic Embeddings; Extending Multi-modal Contrastive Representations) explains why image and text representations stay imperfectly aligned. One consequence is that routes through different modalities make partly different errors. We exploit that difference. A visual knowledge-graph lookup and an OCR-plus-text-search route are close to node-disjoint, so their agreement carries more information than two samples from the same route.
Research Program
The system is built on an open vision-language backbone, Qwen2.5-VL 7B, the backbone used by MMhops-R1. That paper casts multimodal multi-hop question answering as an agent loop that calls image search and text search tools. We adopt the same tools and add two more: an OCR call and a region-crop re-recognition call.
At each turn, the agent writes every tool call and intermediate conclusion as a node in a persistent conversation graph, with edges recording which earlier nodes each step consumed. Edges can point back to nodes from earlier turns, so a turn-three question about opening hours links to the turn-one entity node.
For each question, the agent then runs $K$ chains under a diversity schedule. Chains are not merely resampled. Each one is steered to start from a different perception route: a whole-image lookup, a region crop, OCR, or an entity named by the user. Chains may also reuse cached nodes, and when they do, the reuse is recorded in the graph. We compute $s(a)$ by finding the largest set of node-disjoint chains for each answer. With $K \le 8$ this is a small exact search.
The second component is cross-turn repair. If at least two disjoint fresh chains contradict an inherited node, the system marks that node as superseded and recomputes the dependent nodes. This targets the 27% early-stop cascade directly.
The analysis has three parts. First, we measure how often wrong answers that all samples agreed on share a single upstream node, and which node type it is. Our hypothesis is that it is mostly perception on egocentric images. Second, we fit risk-coverage curves for each confidence signal. Third, we run a small annotation audit that closes the loop to human labels. When node-disjoint consensus confidently contradicts a benchmark's reference answer, two expert adjudicators who have not seen the system output re-judge the item. This tells us how often the reference is wrong, which is the "ground-truth trap" identified for VisDial in Visual Dialogue without Vision or Dialogue. The two adjudicators are drawn from different labs, so their agreement is not an artifact of a shared private rulebook of the kind documented in Common Law Annotations.
Baselines
The naive reference point is the backbone agent answering once with no abstention, scored under the same ternary truthfulness metric. This shows how much of any gain comes simply from being allowed to abstain.
The strongest existing reported system on the main benchmark consists of the frontier models evaluated in CRAG-MM. GPT-5 and Gemini-2.5-Flash reach about 32% truthfulness on single-turn questions, and multi-turn truthfulness is reported at 43% (CRAG-MM). We cannot rerun these at matched compute, so they serve as context rather than as the controlled comparison.
The controlled strongest competitor is plain vote-share abstention. It uses the same backbone, the same tools, the same $K$ chains with the same diversity schedule, and the same development-set tuning of $\tau$, but it counts every agreeing chain equally. This is the baseline most likely to match us, because it already captures much of the benefit of sampling. The only difference between it and our method is whether shared nodes discount agreement, so the comparison isolates the key ingredient.
Two further confidence signals are run at matched calls. One is the model's verbalized confidence. The other is token-level uncertainty of the kind DRAGIN uses to decide when to retrieve, here repurposed for when to abstain.
On MMhops, the reference agent is MMhops-R1 itself, which reports 51.35% bridging accuracy and 22.01% comparison accuracy. On MuMu-QA, the reference is CMT-RAG at 41.73 exact match on the long-dialogue split.
Beating the controlled baseline means two things. The first is a strictly lower area under the risk-coverage curve on CRAG-MM egocentric images, with a paired bootstrap interval that excludes zero. The second is higher truthfulness at the tuned threshold. A secondary win is a lower multi-turn early-stop rate from the repair step.
Two ablations complete the set. One removes the diversity schedule, so chains are only resampled. The other removes cross-turn edges, so inherited nodes are invisible to the disjointness check.
Why not just…?
Why not just use vote share across samples? This is the main threat, and the proposal is designed so that it either wins or clearly loses to vote share. Our mechanism predicts a specific pattern. Vote share and disjoint agreement should be nearly identical on curated images and text-only MuMu-QA, where shared upstream errors are rare. They should separate on egocentric and blurred CRAG-MM images, where one misrecognition is inherited by every chain. If no gap appears on the egocentric split, the claim is wrong and we will report that.
Why not just diversify the inputs and vote, without any graph? Cropping and re-querying the image alone may capture most of the gain, and this is a real competitor. The graph adds two things that input augmentation does not. It tracks reuse of cached nodes from earlier turns, which augmentation at the current turn cannot see. It also supports targeted repair of those nodes. The no-cross-turn-edges ablation separates the two contributions. If that ablation performs as well as the full method on multi-turn sessions, the graph is unnecessary and we would recommend the simpler recipe.
Why not just improve perception or hand everything to a long-context model? Better recognition reduces shared errors but does not tell the system when they remain. The file reports that frontier long-context models still sit at roughly 32–43% truthfulness on CRAG-MM, with heavy multi-turn drift. DocHop-QA reports that adding page images can lower multi-hop accuracy. Neither result suggests that scale removes the need to know when to abstain. As a test, we swap in a stronger perception module under both aggregation rules. If the advantage of disjoint agreement shrinks to nothing, perception quality was the whole story.
Why not just let agents debate to consensus? The evidence in Voting or Consensus? shows that consensus collapses on reasoning tasks, at 28.4% against 54.8% for voting on MuSR. Debate also creates exactly the shared dependence we are trying to discount. We therefore include a debate-to-consensus variant only as a negative reference.
Proposed Experiments
The primary benchmark is CRAG-MM, using single-turn triplets and two- to six-turn conversations. Results are reported separately for curated, egocentric, and noisy images. Metrics are ternary truthfulness, hallucination rate, abstention rate, risk-coverage area, and the early-stop rate on multi-turn sessions.
The decisive check is the interaction effect. The gain over vote share should be largest on egocentric and noisy images and near zero on curated ones. In parallel, the share of confidently wrong answers that trace to a single perception node should be highest on the same split. Finding a uniform gain across splits would undercut the mechanism, even if the method looked good on average.
MMhops tests whether the effect holds on three- to four-hop bridging chains, where a wrong first-hop entity propagates within a single turn. MuMu-QA serves as a text-only control with no perception node, where we predict little gain. The expert audit of confident disagreements with references reports the fraction of reference labels overturned, together with the adjudicators' agreement, as an estimate of label error on these benchmarks.
All conditions use the same backbone, the same $K$, and the same tuning budget, with three seeds.
References
- Crowd Prefers the Middle Path: A New IAA Metric for Crowdsourcing Reveals Turker Biases in Query Segmentation
- Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
- Muse-ing on the Impact of Utterance Ordering on Crowdsourced Emotion Annotations
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- CMT-RAG: Complementary Memory Traces for Multi-turn Multi-hop RAG
- DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation
- GoG: Relation-aware Graph-over-Graph Network for Visual Dialog
- CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmark
- Learning Robust Visual-Semantic Embeddings
- Extending Multi-modal Contrastive Representations
- MMhops-R1: Multimodal Multi-hop Reasoning
- Visual Dialogue without Vision or Dialogue
- Common Law Annotations: Investigating the Stability of Dialog System Output Annotations
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections