Catching Visual Misreadings Before They Cascade

When a vision-language model answers a question that needs several reasoning steps, the chain usually starts with something it has to see. It might read a logo, identify a uniform, or name a landmark. Only then can it look up facts about that entity. If the first reading is wrong, every later step is reasoned correctly about the wrong thing, and the final answer comes out fluent, confident and false. The multimodal claim-verification benchmark MMCV documents exactly this pattern. On 3- and 4-hop claims, models are more confident when they are wrong, and early visual misidentifications propagate through every hop (Piecing It All Together: Verifying Multi-Hop Multimodal Claims).

We propose to catch these misreadings at the step where they happen and to keep alternatives alive rather than commit to one guess. The question is first split into a small tree. Leaves ask purely visual questions, inner nodes ask retrieval or aggregation questions. At each visual leaf, the model answers several times, under light image perturbations and sampling noise. The answers are grouped by meaning. If they fall into one tight group, the answer passes upward. If they split into several groups, the leaf is flagged as doubtful, and the top two readings each spawn their own retrieval sub-tree. Retrieved evidence then decides between them. In short, we use the spread of meaning-level disagreement at visual leaves both to detect the start of a cascade and to decide where to spend extra branches.

Core Claim

Our central claim is about where uncertainty is measured and what is done with it. Prior systems measure confidence at the final answer, or gate tree nodes on token probabilities. Both signals are poorly calibrated at depth in multimodal chains, as MMCV shows. We claim three things. First, disagreement among meaning-clustered answers to isolated perceptual sub-questions detects the initial visual error far better than token likelihood does. Second, branching on the competing readings and letting retrieved text adjudicate recovers many of these errors. Third, at matched compute, this beats both whole-chain majority voting and per-token anti-hallucination decoding on multimodal multi-hop benchmarks.

We define the doubt signal at a leaf as follows. Let $m_1,\dots,m_K$ be the fractions of the $N$ sampled answers that land in each of $K$ meaning clusters. The leaf's dispersion is the entropy

$$D = -\sum_{k=1}^{K} m_k \log m_k,$$

which is zero when all samples agree and grows as answers split. A leaf branches when $D$ exceeds a threshold $\tau$ and the second cluster holds at least mass $\rho$. A global cap $B$ on live hypotheses keeps the tree from exploding.

Two caveats on scope. Grouping sampled outputs by meaning is a simple and generic idea, and we do not claim the estimator itself. The contribution is applying it at decomposed perceptual leaves, mixing image perturbations into the sampling, and turning clusters into evidence-resolved branches.

How This Sits Against Prior Work

Three lines of work motivate the proposal by documenting the failure.

MMCV is a 1–4 hop claim benchmark over text, tables and images. It shows the complexity gap, about 27% below humans at 4 hops, and the overconfidence-when-wrong pattern. It proposes no mitigation.

MIRAGE separates perception errors from reasoning-induced hallucinations on single images (MIRAGE). It finds that spatial hallucinations do not shrink with scale. It also finds that manually correcting intermediate chains lets models reach the right answer about 70% of the time. Both findings suggest intermediate errors are the lever, though MIRAGE uses no retrieval and no trees.

MMhops-R1 trains Qwen2.5-VL-7B as an agent that issues image searches and text searches in a flat turn-by-turn loop (MMhops-R1). It has no explicit decomposition and no mechanism for holding competing visual readings. Its 22.01% on Comparison tasks shows how badly flat agents track multi-image state.

A second group supplies the decomposition machinery, and we reuse it.

Human-guided decomposition into atomic sub-questions corrected over 60% of errors on complex originals in text (Is a Question Decomposition Unit All We Need?). This is the reason to isolate perceptual leaves.

ProbTree builds a query tree and gates each node between closed-book answering, retrieval and child aggregation using token log-likelihood (ProbTree). ProbTree is the closest design to ours. Our difference is that we replace its likelihood gate with meaning-level dispersion under visual perturbation, and we branch instead of choosing a single answer. ProbTree is text-only.

RoHT routes tree nodes between knowledge bases and text (RoHT). It is also text-only.

G²-Reader is the strongest multimodal tree system. It maintains a planning graph over document content with an evidence-checking agent (G²-Reader). It degrades beyond 3 refinement rounds because it adds redundant sub-questions. Our branching is targeted by the doubt signal rather than expanded greedily, which we expect to avoid that failure.

A third group constrains the design. Single-turn hallucination fixes include RITUAL (RITUAL), SHARP's steering vectors (SHARP), Reefknot's entropy-triggered calibration (Reefknot) and HalTrapper's context-aware decoding (Why LVLMs Are More Prone to Hallucinations in Longer Responses).

RITUAL contrastively decodes against random image transformations. We borrow its observation that perturbations expose unstable visual readings, but we use the perturbations to measure disagreement, not to reweight tokens. HalTrapper shows that long rationales drift toward attending to generated text instead of the image. This is an argument for answering perceptual leaves in short, fresh contexts with the image re-presented, which our tree does by construction.

Finally, the clustering direction supplies methods but not the setting. Auto-CoT clusters questions with Sentence-BERT to pick demonstrations (Auto-CoT). CoDR clusters multimodal embeddings into a diverse demonstration coreset and reports lower object hallucination (CoDR). Both cluster static data offline. We cluster the model's own test-time answers, per leaf, to read off uncertainty.

Research Program

The system has four parts, all built on one open backbone, Qwen2.5-VL-7B, which matches the MMhops-R1 backbone.

The decomposer prompts the model to emit a query tree. Each leaf is labeled as perceptual (answerable from the image alone), retrieval (answerable from text search), or aggregation. Leaf labels are checked by asking the model whether the leaf mentions any entity not yet resolved.

The leaf sampler answers each perceptual leaf $N$ times. It mixes temperature sampling with crops and color jitter on the image. Flips are excluded, since they invert left and right and would manufacture spatial disagreement.

The clusterer groups the short answers. One variant uses mutual entailment judged by the same model. A cheaper variant uses a Sentence-BERT cosine threshold, following Auto-CoT's embedding choice.

The resolver handles flagged leaves. For each live reading, it runs that reading's downstream retrieval sub-tree. It then asks the model whether the retrieved passages are consistent with both the reading and the image, and keeps the branch with the higher consistency. Ties fall back to cluster mass.

The analysis work matters as much as the system. We will build leaf-level error labels in two ways. Where a benchmark exposes the first-hop entity, we use it directly. Otherwise we hand-annotate a stratified sample of about 500 questions per benchmark for the correctness of the first visual reading. With these labels we can measure three things separately: how often final errors trace to a leaf misreading, how well each uncertainty signal detects those misreadings, and how often branching recovers them.

Baselines

The naive reference point is the backbone answering directly with linear chain-of-thought and no retrieval tree. It tells us how much any structure helps at all.

The strongest existing method on the main benchmark is MMhops-R1. It reports 51.35% accuracy on MMhops Bridging and 22.01% on Comparison, against 36.62% and 8.76% for GPT-4o (MMhops-R1). It is the strongest competitor because it uses the same 7B backbone, has the same search tools, and was trained by reinforcement learning specifically for this benchmark. We will run its released policy, if available, with the same retriever and Wikipedia snapshot as ours. Otherwise we compare against its reported numbers, with that caveat stated. Beating it means exceeding 51.35% on Bridging without any task-specific training, or showing that adding our leaf branching on top of its policy raises both numbers.

On document QA, G²-Reader reports 66.21% average accuracy on VisDoMBench with Qwen3-VL-32B (G²-Reader). We will add our leaf sampler and resolver to a re-implementation at the same backbone and compare against that re-implementation rather than claim parity with the 32B number.

On MMCV, the reported reference is Gemini 1.5 Flash at 70.92% average open-book F1 (MMCV). No open-model mitigation result is reported there. Our target is the 3–4 hop slice, together with a measured reduction in confidence on wrong answers.

The strongest structural competitor is a ProbTree-style tree on our multimodal decomposer that gates leaves by token log-likelihood. It is identical to our system except for the uncertainty signal and the absence of branching, and it is the cleanest test of our claim. For the tree baselines, every comparison is matched on total generated tokens and forward passes, and each method gets the same threshold-tuning budget on a 300-question development split.

Ablations isolate each ingredient. One removes image perturbations, keeping temperature sampling only. One replaces entailment clustering with embedding clustering. One detects without branching, abstaining on or re-querying flagged leaves. One branches uniformly on every perceptual leaf rather than only doubtful ones, which tests whether targeting matters. One gives the same budget to deeper G²-Reader-style refinement.

Why not just…?

Why not just use token log-likelihood at each leaf, as ProbTree does? This is the most direct threat, because leaf questions are short and likelihood may be better calibrated there than at the final answer, where MMCV observed overconfidence. We will settle it directly: on the annotated leaf labels, compare the area under the detection curve of likelihood against meaning dispersion. If likelihood is as good, we keep the branching mechanism but adopt the cheaper signal. The claim would then shrink to "branch on doubtful leaves," and we would report it that way.

Why not just sample whole chains and take a majority vote? Voting is simple and spends compute on the same goal. But when the visual misreading is systematic, most chains share it, and the vote ratifies the error. Our sampling is concentrated where the uncertainty originates, and the competing readings are adjudicated by evidence rather than by headcount. No reported result was found on Lacuna for self-consistency voting on MMhops, MMCV or VisDoMBench. We will therefore run it at equal token budget. If it matches us, the tree adds nothing beyond cost, and the proposal fails.

Why not just apply RITUAL or SHARP at every step of an existing agent? These methods improve single-turn perception. RITUAL raises LLaVA-1.5 POPE accuracy from 84.13% to 88.87%, and SHARP raises LLaVA-Next adversarial POPE from 79.5% to 82.8%. That makes them a fair complementary baseline. However, they output one sharpened answer per step and cannot recover when the visual evidence is truly ambiguous and only retrieved text can disambiguate. We will run both inside the same tree. Combining SHARP with our branching is also a natural variant, since the two act at different levels.

Why not just cap the tree depth to avoid G²-Reader's over-refinement? Capping stops the redundancy but spends nothing on the leaves that are actually wrong. The uniform-branching and deeper-refinement ablations show whether targeted branching beats untargeted extra compute.

Decisive Experiments

The first check is diagnostic. On annotated MMhops Bridging and MMCV 3–4 hop samples, what fraction of final errors begin with a leaf misreading? If it is small, say under a third, the premise fails and we stop.

The second check compares leaf error detection by meaning dispersion against token likelihood. Success means a clear and consistent gain in detection quality across both benchmarks. MIRAGE's spatial category serves as a stress test, since its spatial errors are expected to be stable under perturbation and thus hard to detect.

The third check is end-to-end accuracy at matched compute. It runs on MMhops Bridging and Comparison, the MMCV 3–4 hop slice, and VisDoMBench (including SlideVQA, where G²-Reader reports a gain from 40% to 74%), against the baselines above. We report the recovery rate, meaning the fraction of flagged-and-wrong leaves that branching corrects. On MMCV we also report calibration, testing whether the overconfidence-when-wrong curve flattens.

The proposal holds only if three conditions are met together: leaf misreadings explain a large share of errors, dispersion detects them better than likelihood, and branching beats voting at equal cost.