When a vision-language model (a language model that also reads images) describes a sequence of pictures or video frames, it does more than name objects. It links events with words like "because," "but," "then," and "so that." Each of these words is a claim about how two scenes relate, and such claims are often ambiguous. Human readers given the same pair of sentences frequently disagree on whether the link is a cause or just a sequence, a contrast or a concession. We propose that many hallucinations in multi-image descriptions (fluent statements the images do not support) begin at these connecting words. The model commits early to the most familiar story, for example "the glass fell, so it broke," and then invents visual details to justify that story.
Our method treats the relation between two visual events as an explicit uncertain belief. Before committing to a connective, the model asks the images a small number of short questions. Each question is chosen to best separate the remaining candidate relations, in the sense of expected information gain, the expected drop in uncertainty from hearing the answer. The model then speaks with the confidence that its updated belief warrants. Measuring success requires a resource that does not yet exist, image and video-frame pairs labeled with crowd-sourced distributions over discourse relations. The central test is whether querying moves the model's belief toward the human distribution rather than toward false certainty, and whether doing so removes the invented details that follow strong connectives.
Core Claim
The claim has two parts. First, hallucinated visual details in multi-scene narration concentrate right after relational commitments such as causal or contrastive connectives. A model's uncertainty over the relation between two scenes predicts where those fabrications will appear. Second, choosing visual questions by expected information gain over a set of relation hypotheses has two measurable effects. It reduces those fabrications. It also brings the model's relation distribution closer to human soft labels. The second effect is the distinctive prediction: the model should shrink only the uncertainty that the pixels can resolve, and leave in place the interpretive ambiguity that humans themselves retain.
The proposal draws on the three directions in different roles. Bayesian query selection supplies the mechanism. Discourse relation ambiguity defines the hypothesis space and the evaluation target. Multimodal hallucination is the failure we aim to reduce and the regime we measure in.
Where this sits relative to prior work
Several papers motivate the gap. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that in long outputs, attention drifts from image tokens to the model's own prior text as it tries to stay coherent. It responds by suppressing context with contrastive decoding, but never asks which coherence relations cause the drift. Mitigating Hallucinations in LVLMs via Summary-Guided Decoding explicitly leaves verbs and conjunctions unguided, treating them as harmless glue. We treat exactly those tokens as the commitment points. Fine-Grained Multi Image Object Hallucination Benchmark shows that presenting several images together sharply degrades even object existence judgments, but it probes only object facts with static multiple-choice questions, not relations between images.
Other work constrains the design. DiscoGeM 2.0 collects implicit discourse relations from 10 annotators per item via connective insertion. It finds stable co-occurring pairs such as Contrast/Concession and Cause/Level-of-Detail, and an intra-lingual Jensen-Shannon divergence (JSD) of 0.43 between annotator distributions. Its lesson is that a single "correct" relation is often the wrong target. We're Afraid Language Models Aren't Modeling Ambiguity and Zero and Few-shot Semantic Parsing with Ambiguous Inputs show that zero-shot models collapse to one reading: GPT-4 disambiguates only 32% of the time, and zero-shot top-5 coverage of both readings is near 0. These results argue against trusting the model's raw relation distribution, so our method should not simply read off the model's first preference.
Three lines of work supply the method. BED-LLM contributes two techniques we reuse. It scores candidate questions by expected information gain using prior-likelihood pairing, meaning it asks how likely each answer is under each hypothesis. It also shows that maximizing answer entropy alone selects noisy, unanswerable questions. CA-BED contributes soft Bayesian updates, which avoid wrongly eliminating hypotheses when answers are noisy. Both operate on text-only entity guessing games with discrete targets, and CA-BED names discrete hypothesis sets as a limitation. Discourse relations are a small, fixed inventory, which suits these methods, but the target is a distribution, not a single entity. That change in target is our main departure.
The Perceptual Bandwidth Bottleneck in Vision-Language Models (FOVEA) is the closest multimodal neighbor. It frames choosing image crops as sequential experimental design over a single image. However, it assumes that a good view guarantees a correct answer, and its queries concern where to look, not which interpretation holds. ViThinker learns when to call internal vision experts under a sparsity penalty, without an explicit belief. Visual Discourse Parsing is the only paper here that predicts rhetorical relations between video scenes. It uses an LSTM with single-label trees, which is precisely the single-label assumption DiscoGeM argues against.
Research Program
Benchmark. We will build a soft-label multimodal discourse set. Source pairs come from consecutive keyframe pairs in the 310 verified videos of Visual Discourse Parsing, targeting roughly two thousand pairs. Following DiscoGeM's one-step connective insertion protocol, 10 crowd workers see two short neutral captions, each restricted to what is visible in one frame, alongside the frames. Each worker drags in the connective they find most natural, which yields a probability distribution over relations per pair.
We add a control set of about five hundred pairs drawn from unrelated images in MIOH's COCO-based image sets. For these pairs the right answer is a weak relation such as plain juxtaposition, so any confident causal story is a relational hallucination by construction. For a subset of pairs we also collect a second annotation round with the frames hidden. The difference between the two rounds tells us, per pair, how much of the human uncertainty is visually resolvable. That quantity is what the method is supposed to remove.
Method. Let $r$ range over a fixed relation inventory: DiscoGeM's level-2 senses plus a "no substantive relation" option. The open model Qwen2-VL-7B, the strongest open-weights model on MIOH, plays three roles.
It first provides a prior $p(r)$ by scoring connective insertions given the frames. It then proposes candidate closed-form visual questions $q$, such as "is the liquid on the floor in frame 2 the same color as the liquid in the glass in frame 1?" For each hypothesis it predicts an answer distribution $p(a \mid q, r)$, using BED-LLM's prior-likelihood pairing. We select
$$q^* = \arg\max_q \; H\big(p(r)\big) - \mathbb{E}_{a \sim p(a \mid q)}\, H\big(p(r \mid q, a)\big),$$
where $H$ is Shannon entropy and $p(a \mid q) = \sum_r p(a \mid q, r)\,p(r)$ is the predicted answer distribution.
The question is then answered by the same model in a fresh, short prompt containing only the relevant frame or crop. This deliberately avoids the long-context drift identified by HalTrapper. The answer's probability vector $\pi(a)$ enters a soft update:
$$p(r \mid q, a) \propto p(r) \sum_a \pi(a)\, p(a \mid q, r).$$
We stop when the best remaining expected information gain falls below a threshold or after a budget of $k$ questions. Narration then uses the posterior in two ways. The connective is chosen from the posterior, with a hedged two-relation realization if posterior entropy stays high. Content supporting the relation is restricted to facts established by the answered questions.
Analyses. The first analysis tests the localization claim. Using human-verified hallucination annotations on baseline narrations, we ask whether fabricated details cluster in the clause following a strong connective, and whether the model's pre-query relation entropy predicts them. The second analysis tests calibration. We measure JSD between model and human distributions before and after querying, separately for pairs whose uncertainty is visually resolvable and pairs whose uncertainty is not, as identified by the frames-hidden round. As a text-only sanity check, the prior and update machinery is also run on DiscoGeM 2.0 English text. This confirms that the model's relation scores can track human soft labels at all before images enter.
Baselines
The naive reference is direct narration. The same model describes the two frames with no querying, and its connective distribution is read from token scores. The file has no reported result on a multimodal soft-label discourse task, since none exists. For single-label scene relations, the reported reference is 0.63 Relations Accuracy on the 310-video set, from an attention LSTM in Visual Discourse Parsing. We will report majority-label accuracy on our pairs for comparability, while treating JSD as primary.
The strongest hallucination competitor is Woodpecker. It extracts concepts from a draft, asks verification questions about each, checks them with Grounding DINO and BLIP-2, and rewrites the draft. It reports lifting MiniGPT-4's POPE accuracy from 54.67% to 85.33%, with about a 6.2% mis-correction rate. We run it on the same narrations, swapping in the same Qwen2-VL-7B as its question answerer so the perception quality matches, and we cap its question count at the same budget $k$.
We also include Volcano-style critique and revise prompting, matched for forward passes. Volcano reports 88.3% overall POPE accuracy, but it requires fine-tuning. We use its prompting form, and note that this is weaker than the trained system. HalTrapper's contrastive contextual decoding is included as the decoding-time competitor, for which the file gives no numeric CHAIR values.
The query-selection competitor is a CA-BED-style controller with soft updates but no visual answering. Here "answers" come from the model reasoning over the full multi-image context, which isolates whether the fresh short-prompt look at the image matters.
Two ablations isolate the key ingredients. Random-question selection draws from the same candidate pool with the same budget, which isolates the information-gain criterion. Marginal-entropy selection follows BED-LLM's warning about noisy questions.
All methods use the same model, the same candidate-question generator, and the same number of model calls per pair. Thresholds are tuned on a shared development split with identical search budgets.
Beating the competitors means four things together. Our method should achieve lower relational and object hallucination rates than Woodpecker at equal call budget. It should show lower JSD to human distributions than direct narration and the text-only belief controller. And it should outperform random-question selection by a margin larger than the seed-to-seed spread.
Why not just…?
Why not just sample many narrations and use their connective frequencies as the soft distribution? Self-consistency sampling is cheap and may well calibrate the spread better than one greedy pass. However, every sample draws on the same language prior. The AmP and AMBIENT results show that this prior collapses to one reading, so sampling cannot introduce visual evidence the model never inspected. We include sampling with matched call budget. If it matches our method on JSD on the visually resolvable subset, the querying claim fails on that axis.
Why not just hedge whenever the model is uncertain, using "and then" instead of "because"? This is the most serious threat. Much of the measured hallucination reduction could come from making weaker claims rather than from looking. We run an entropy-threshold hedging baseline with no questions, tuned to match our method's rate of strong connectives. If querying helps, it must show two things this baseline cannot. It must pick the human-preferred strong relation more often on resolvable pairs. It must also keep strong connectives where humans use them, rather than trading informativeness for safety.
Why not just run Woodpecker-style verification on every noun? Noun-level checking repairs object words but not the relation itself. The reasoning-gap findings of Yan et al. suggest that a fabricated causal story corrupts what follows. In addition, checking every noun spends the question budget on facts that do not discriminate between relations. The matched-budget comparison will show whether targeted relational questions buy more per call. If Woodpecker wins on object hallucination but loses on relation JSD, that is a meaningful split we will report.
Why not just give the model all frames at higher resolution, or a longer context? MIOH reports that existence accuracy drops to about 30% at eight images, and HalTrapper reports that richer context brings hallucinations earlier. Both suggest that more context makes the problem worse. Our fresh-prompt answering ablation tests this directly.
Decisive Checks
Three outcomes would settle the proposal.
The first is localization. If fabricated details are not more frequent after strong connectives than elsewhere, and pre-query relation entropy does not predict them, then the premise is wrong and the work reduces to a generic verification method.
The second is the calibration signature. On visually resolvable pairs, querying should lower model entropy and JSD to the human distribution. On interpretively ambiguous pairs such as Contrast versus Concession, posterior entropy should stay near the human entropy rather than fall toward zero. The question-selection criterion should also assign low expected gain to these pairs, which is a prediction no existing baseline makes.
The third concerns the unrelated-image control set. The method should move probability onto "no substantive relation" and eliminate confident causal claims there. The hedging baseline, by contrast, should only soften them without consistently identifying which pairs are unrelated.
We will also report POPE and CHAIR on single images to confirm that the procedure does not degrade ordinary grounding, along with per-pair latency against Volcano's reported increase from 2.7s to 5.8s.
References
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Mitigating Hallucinations in LVLMs via Summary-Guided Decoding
- Fine-Grained Multi Image Object Hallucination Benchmark
- DiscoGeM 2.0: A Parallel Corpus of English, German, French and Czech Implicit Discourse Relations
- We're Afraid Language Models Aren't Modeling Ambiguity
- Zero and Few-shot Semantic Parsing with Ambiguous Inputs
- BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design
- CA-BED: Conversation-Aware Bayesian Experimental Design
- The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design
- ViThinker: Active Vision-Language Reasoning via Dynamic Perceptual Querying
- Visual Discourse Parsing
- Woodpecker: hallucination correction for multimodal large language models
- Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision
- Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models