When several vision-language models work together, each often sees only part of the scene. Examples include multiple cameras, image tiles, or drones covering different ground. Sometimes an agent should believe a peer's claim about something it cannot see, because the peer genuinely saw it. Sometimes it should reject the claim, because the peer invented it. Current anti-hallucination methods handle only one of these cases. They push every model back toward its own image, which is correct when the peer is wrong and harmful when the peer saw something the listener could not.
We propose a simple rule taken from formal work on trust. A claim should count as evidence only from agents whose view actually covers the region the claim is about. Each first-hand observation should count once, however many times it is repeated. Each peer's weight should be learned from how often its past claims matched what the listener could check for itself. We test whether this rule lets a group of partial-view agents approach the object-hallucination accuracy of a single model that sees the whole image. We also test whether it stops the cascading errors that current multi-agent systems show.
Core Claim
In multi-agent vision-language systems, hallucination spreading is largely an epistemic accounting failure rather than only a perception failure. Agents count a peer's words as evidence regardless of whether that peer could have seen the thing described. They also re-count claims that have been relayed through several agents. Tagging every claim with its original source and the image region it refers to, and gating belief updates on who could see that region, should fix most of the problem. It should also avoid the cost that "trust your own eyes" methods pay whenever evidence lies outside the listener's view.
The trust literature supplies the method. Trust-Sensitive Belief Revision replaces "always accept new information" with trust tied to domain-specific competence. In that paper, an informant is believed only about the parts of the world it is competent on, represented as cells of a partition. We make those cells concrete and geometric: an agent is competent about exactly the regions its crop covers. Two further papers supply the provenance machinery. On the Formal Semantics of Speech-Act Based Communication in an Agent-Oriented Programming Language tags every belief with its origin (percept, self, or a named agent) and separates receiving a message from adopting it. In Data We Trust: The Logic of Trust-Based Beliefs separates believing a proposition from believing it because a trusted source said so. We reuse both ideas directly.
Hunter and Booth leave open how trust should change over time; their partitions are static. We answer that with the simplest possible dynamic: trust is estimated from agreement on regions both agents can see. The Lock-in Hypothesis: Stagnation by Algorithm motivates the rule against double counting. It shows that mutual trust above a threshold drives networks to false consensus because ungrounded claims are counted repeatedly. That analysis is text-only; we test whether the same mechanism operates when the claims are about images.
What Prior Multimodal Work Does and Does Not Do
The closest work is Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow (ViF). It shows that downstream agents adopt an upstream agent's visual error, and that attention to image tokens falls by up to 60% by turn 20. Its fix is to relay attention-selected visual tokens between agents and to amplify image attention in middle layers. ViF assumes every agent sees the same image and has no model of who is reliable about what. It motivates our problem, but its mechanism cannot help when the relevant evidence is off-screen for the listener.
Flag Game: Interpreting Decision Mechanisms of Bounded Social Agents is the only work with genuinely partial views. There, agents see 6x4-pixel crops of one of 28 flags. The paper measures how much agents defer to peers and whether they check peer claims against their own crop. It reports that the prompt "treat peer reports as observations of unseen regions" raised group accuracy from 0.54 to 0.81. That result is our strongest motivating evidence: the visibility distinction matters. However, the domain is a toy, the fix is a prompt, and the paper does not test natural images or standard hallucination benchmarks.
Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System resolves conflicts through free-form debate backed by an object detector. It is sensitive to how the supervisor is prompted and has no explicit trust accounting.
Single-model methods act as constraints on our design. Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models makes three findings:
- False premises in dialogue history cause drops of at least 31% for open-source models.
- The flip rate approaches 100% on imaginative premises.
- Its Residual Visual Decoding (RVD) method shifts predictions toward the image-only distribution when the two disagree.
That design choice is exactly what we expect to fail under partial views. There, "the image-only prediction says no" frequently just means "I can't see it."
Research Program
Setting. We build a partial-view object-probing testbed from COCO images, using POPE (Polling-based Object Probing Evaluation). POPE asks yes/no questions of the form "is there an X in the image?" and draws its negatives in three ways: random objects, popular objects, and adversarial objects that commonly co-occur with objects actually present (Evaluating Object Hallucination in Large Vision-Language Models). We also use the COCO-sourced portion of AMBER (An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation), which adds attribute and relation questions.
Each image is cut into K overlapping crops, one per agent, with K between 2 and 6 and overlap between 10% and 40%. COCO bounding boxes give ground truth for which crops contain which objects. The shared coordinate frame is known to all agents, which mirrors multi-camera rigs with calibrated layouts.
Messages. Agents exchange structured claims containing four fields: object (with optional attribute), claimed region cell, first-hand source ID, and the source's own yes-probability. The yes-probability is read from the model's yes/no logits on its crop.
Belief update. Agent $i$ computes its belief in a claim $c$ about region $r_c$ as follows:
$$\ell_i(c) = \ell_0 + \sum_{k \in S(c),\; r_c \in V_k} \tau_{ik}\, \ell_k(c)$$
The terms are defined as:
- $\ell_0$ is a fixed prior log-odds, tuned on a held-out split.
- $S(c)$ is the set of distinct first-hand sources for $c$, deduplicated by source ID, so relayed copies add nothing.
- $V_k$ is the set of region cells visible to agent $k$.
- $\ell_k(c)$ is agent $k$'s own log-odds that $c$ is true from its crop.
- $\tau_{ik}$ is $i$'s trust weight in $k$, with $\tau_{ii}=1$.
The listener's own "no" therefore counts only if the listener can see the region. The trust weight $\tau_{ik}$ is updated across turns and images. It is the posterior mean of a Beta distribution over whether $k$'s claims about jointly visible cells agreed with $i$'s own confident judgments. This is the dynamic, iterated trust that Hunter and Booth leave open. A final LVLM pass writes the answer or caption conditioned only on claims whose combined belief exceeds a threshold.
Analysis thread. Beyond accuracy, we test whether current LVLMs already represent "the peer could have seen this" internally. SHARP: Steering Hallucination in LVLMs via Representation Engineering shows that faithful and hallucinated states are linearly separable in middle layers (layers 10–15). We fit the same kind of linear probe to the listener's hidden state when it reads a peer claim, labeling each claim as in-view versus out-of-view and true versus false. If a visibility direction exists but is ignored at decoding, that locates the failure. This thread is diagnostic only; the method itself does not depend on it.
Baselines
The naive reference point is a group in which each agent sees its crop, reads peer messages as plain dialogue, and answers after three rounds of broadcast. This measures raw snowballing under partial views.
Two single-agent references bracket the problem. The first is one agent on one crop, which gives a floor. The second is one base LLaVA-1.5 on the full image, which gives the target. For that target, RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs reports 84.13% POPE accuracy for base LLaVA-1.5 and 88.87% with RITUAL. Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision reports 88.3% POPE accuracy. We rerun these references ourselves on our COCO subset rather than trusting cross-paper comparability.
The strongest existing multi-agent method is ViF. Its reported numbers are an average gain of 2.4–3.8% across 10 base models and a reduction of at least 30% in its snowballing score (near 40% in circular topologies). We run ViF as described in that paper, placing each agent on its own crop, with the same base models, rounds, and topologies (linear, layered, random, circular) as our method.
The strongest single-stream decoding competitor is RVD applied per agent. The reported figure is that LLaVA-1.5 accuracy under false-premise conversation rises from about 15% to 33%, from the snowballing paper cited above. The strongest prompt competitor is the Flag Game "unseen regions" prompt, with a reported 0.54→0.81 on Flag Game. We also run a debate-with-detector pipeline in the style of the post-hoc multi-agent paper, as a costly upper reference.
Fairness is enforced in four ways:
- All methods use the same base models: LLaVA-1.5 7B and 13B, and LLaVA-NeXT.
- All methods use the same crops and number of communication rounds.
- Every tunable threshold gets the same held-out tuning budget of 200 images.
- Latency is measured on the same hardware.
What counts as beating the baselines. Our method beats them if it reaches the following on POPE-adversarial collective accuracy with $K\ge 3$:
- a statistically significant margin over ViF and RVD, with paired bootstrap over images;
- a lower flip rate on correct beliefs;
- a higher acceptance rate of true out-of-view reports.
Ablations. Four ablations isolate the key ingredients:
- removing the visibility gate, so every agent's opinion counts on every claim;
- removing source deduplication;
- freezing trust at 1;
- replacing learned trust with oracle per-agent accuracy.
Why not just…?
Why not just revert each agent to its own image when dialogue and vision disagree, as RVD does? Under shared full views this is a strong fix, and we expect RVD to match us there. Under partial views, the image-only prediction for an out-of-view object is a confident "no" driven by what the crop lacks. RVD would then reject true peer reports. Our testbed separates in-view claims from out-of-view claims. The prediction is that RVD wins or ties on in-view claims and loses on out-of-view true claims. If RVD does not lose there, the visibility gate is unnecessary and we will say so.
Why not just prompt agents to treat peer reports as observations of unseen regions? This is the most serious threat, because it expresses the same idea in words and reported a large gain on Flag Game. The file also documents two weaknesses. Prompt fixes are brittle, as the post-hoc multi-agent paper found for supervisor prompts. Weaker models override their own evidence regardless, as with the "sycophantic override" archetype in Flag Game and the near-99% "yes" rates on POPE. We run the prompt across five paraphrases and all base models and report the variance. If the prompt matches our method on large models, the remaining contribution shrinks to small models plus the deduplication and trust-learning ablations. We would report that outcome plainly.
Why not just have each agent answer independently and take a vote? Independent answers avoid snowballing, but majority vote throws away the fact that often only one agent can see the object. An "any agent says yes" rule avoids that problem but inherits every agent's yes-bias. The honest version of voting knows which agents can see which region, and that is our method without trust learning. We therefore include plain majority vote, any-yes pooling, and visibility-aware pooling with uniform weights. The last of these is the ablation that shows what learned trust adds.
Why not just give every agent the full image and use ViF? In many deployments the views are genuinely partial: different cameras, or bandwidth-limited tiles. In the shared-view setting we still run a decisive check (see below) to learn whether deduplication helps beyond attention forcing.
Proposed Experiments and Decisive Checks
Main results. We report POPE collective accuracy, F1, and yes-rate for each negative-sampling setting and each K. Alongside these we report AMBER existence, attribute, and relation scores and CHAIR on final captions. CHAIR measures the fraction of captioned objects, or of caption sentences, that mention objects absent from the image.
Hostile peer. To test robustness to hallucinating peers, we inject one hostile agent. This agent either receives an MMHalSnowball-style false premise (Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models) or is a weaker model. We then measure how quickly $\tau$ for that peer falls and how much of its false content reaches final answers.
Manipulability. Hunter and Booth note that trust-sensitive operators can be manipulated. We therefore run a peer that is honest on overlap cells but lies about regions only it can see. We expect our method to fail here and will quantify the failure, since the trust rule cannot detect lies in unverifiable regions.
Lock-in versus attention. This decisive mechanistic check separates the Lock-in account from the ViF account. With all agents sharing the full image in a circular topology, we compare four conditions: deduplication alone, ViF alone, both, and neither. If deduplication alone removes a substantial share of the snowballing, the failure is partly double counting rather than purely attention decay. If only ViF helps, the attention account stands.
Collective accuracy versus full view. The second decisive check asks whether a group of three to six agents with gated trust closes most of the gap between single-crop and full-image POPE accuracy, and does so at lower latency than Volcano-style revision loops.
Transfer to Flag Game. Finally, we replicate the method on Flag Game itself and compare against its reported 0.81 prompt result, to confirm that the rule transfers back to the setting where the visibility effect was first observed.
References
- Trust-Sensitive Belief Revision
- On the Formal Semantics of Speech-Act Based Communication in an Agent-Oriented Programming Language
- In Data We Trust: The Logic of Trust-Based Beliefs
- The Lock-in Hypothesis: Stagnation by Algorithm
- Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
- Flag Game: Interpreting Decision Mechanisms of Bounded Social Agents
- Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System
- Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
- Evaluating Object Hallucination in Large Vision-Language Models
- An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs
- Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision