Models that take sound, video and text together often hallucinate. They describe what they have heard or seen, and they fill in plausible details that are not there. Two bodies of evidence point in opposite directions on this. Affective computing finds that for questions about how someone feels, the voice is the most reliable channel and raw video is the noisiest; audio alone predicts arousal far better than video alone. Multimodal foundation models do the reverse. They lean on vision and on language priors and ignore conflicting audio. This proposal turns that mismatch into a design rule. When a system makes a factual claim about a clip, the claim should be checked by the sensory channel that has the authority to settle it. The question of which channel holds that authority should be measured for each kind of claim, not fixed in advance.
Concretely, we break a generated answer into small, checkable claims ("the speaker sounds angry," "a dog is visible," "the word 'great' was said sarcastically"). We sort each claim by type and send it to a verifier that sees only one modality. A fixed table, estimated from held-out data, decides whose verdict wins for that claim type. Several instances of the same model each play one role, which makes this a multi-agent system. The structure is deliberately narrow, though: the agents answer yes/no questions and never debate in open text. The aim is to fix hallucination in the cases where channels disagree, such as sarcasm, masked emotion, or a sound with no visible source. Those are the cases where today's models fail most and where simple ensembling should help least.
Core Claim
Our hypothesis is a claim-type-conditional trust table, which we call the jurisdiction map. It records, for each claim type and each modality, how reliable that modality's isolated verdict is. Routing verification through this map will reduce cross-modal hallucination more than three alternatives: uniform voting across modality-isolated verifiers, single-model contrastive decoding, and captioning everything to text. We also predict the gains will sit almost entirely in cases where the modalities disagree. For affective and prosodic claims, we expect the estimated map to rediscover the human acoustic-primacy ordering, even though the underlying model's joint-input behaviour is visually dominated.
Formally, let $c$ be a claim type and $m \in \{\text{audio}, \text{video}, \text{transcript}\}$ a modality. Let $r_{c,m}$ be the accuracy of a verifier that sees only $m$ on calibration claims of type $c$, and let $v_m \in \{-1, +1\}$ be that verifier's verdict on a new claim. The arbiter keeps the claim when
$$ \sum_{m} \log\frac{r_{c,m}}{1-r_{c,m}} \, v_m > \tau, $$
where $\tau$ sets the trade-off between abstaining and asserting. This is the standard weighted vote for independent voters with known accuracies. The weight a modality receives depends on the claim type, not on the clip. As a result, one highly reliable channel can overrule two weak ones.
This differs from the closest prior work in three ways. XGC-AVis splits audio-visual question answering into perception, planning, execution and a final decider that reconciles disagreeing executors. It treats audio only as transcripts and sound captions, and its authors say emotional cues remain unsolved. Its decider settles disagreements between answers, not between modalities. Visual Flow shows that agents chatting in text gradually stop attending to the image, so early errors snowball. It repairs this by passing visual tokens between agents, but only for vision and text. We take its diagnosis, that downstream agents treat upstream text as truth, and apply it across modalities: no claim travels downstream until its jurisdictional modality has signed off on it. AVCD contrasts a model's outputs with and without each modality masked, and does so inside a single decoding pass. It triggers only when the model is uncertain, so it skips confidently wrong outputs. Our verification runs on every claim regardless of the generator's confidence.
Research Program
The work has four parts.
Measuring the hierarchy inside a model. For each base model, we run the same model three times on each calibration item: audio only, video frames only, and transcript only. We score its yes/no verdicts on atomic claims, which gives the jurisdiction map. The diagnostic findings motivate this step. AVHBench shows audio-visual models often do worse with both inputs than with one, its "more is less" result. The Curse of Multi-Modalities attributes failures to dominance of a single modality and to spurious sound-object correlations. The affective literature supplies the prior we expect the map to recover. On AVEC2015, Conditional Attention Fusion reports arousal concordance of 0.787 from audio versus 0.432 from video. The detection of emphatic words finds a text-only emphasis detector at 79.7%, below the 79.8% majority baseline, while an acoustic-only detector reaches 87.1%. Claim types will come from a fixed taxonomy: object presence, sound source, spoken content, emotion and arousal, emphasis and prosody, and speaker intent including sarcasm. A text model with a small labelled set assigns each claim to a type.
Building the system. A generator, the unmodified omni model with all inputs, writes an answer or description. A decomposer splits it into atomic claims and types them. Three modality-isolated verifiers judge each claim. The arbiter applies the equation above and either deletes rejected claims or regenerates with the rejection made explicit. All roles use the same base weights. The only difference between roles is what each one sees.
Handling the reasoning trade-off. EmoPrefer found that adding chain-of-thought and external-judge steps hurts strong models on emotion judgments, because errors accumulate across the steps. That finding constrains the design. Verifiers produce one yes/no token, and the arbiter is arithmetic, not a language model. This keeps the number of free-text steps where error can compound to one, the decomposition. We test explicitly whether replacing the arithmetic arbiter with a language-model arbiter reintroduces the degradation EmoPrefer saw.
A new affective-claim test set. Existing hallucination benchmarks use environmental sounds and objects; AVHBench says outright that it leaves speech and prosody aside. We therefore build a claim-level set. We take model-generated descriptions of MUStARD sarcasm clips, IEMOCAP dialogues and T-MED classroom clips, annotate each atomic claim as supported or unsupported, and mark whether the clip's channels agree or conflict.
Baselines
The naive reference is the base omni model given all inputs and answering directly. We use Qwen2.5-Omni, the strongest open model in EmoPrefer and CCFQA, plus VideoLLaMA2 and video-SALMONN, so that we can compare against AVCD on the models it was evaluated with.
The strongest training-free competitor is AVCD, run with its released settings on the same base models. It is the closest method that also uses modality masking to fight hallucination. On AVHBench it reports accuracy gains of 2% for VideoLLaMA2 and 7% for video-SALMONN; these are relative improvements, and the file gives no absolute values. Beating it means higher AVHBench accuracy on both models at matched or lower wall-clock cost, and a larger gain on the subset where the base model's answer was high-confidence.
The strongest trained competitor is LoRA fine-tuning of the base model on the AVHBench training split. AVHBench reports this lifts Video-LLaMA on audio-driven video hallucination from roughly 50% to 83.9%. We give it the same labelled calibration data our map uses, and we report in-distribution and transfer to CMM separately. Matching it in-distribution while transferring better would count as a win. Losing in-distribution but winning on transfer is a partial result.
The multi-agent system competitor is an XGC-AVis-style pipeline: transcript and sound captions, planner, two executors and a decider. We rebuild it on our base model with the same call budget, since XGC-AVis itself used Gemini 2.0 Flash and reports a 6.8% average-accuracy gain over it on XGC-AVQuiz.
Two ablations isolate the key ingredient, and they matter most. The first keeps the same isolated verifiers but combines them with an unweighted majority vote. The second weights each modality by its overall reliability, ignoring claim type. If either matches the full system, the jurisdiction map adds nothing and the gain comes from ensembling alone.
Every method gets the same base weights, the same calibration data where any is used, and a reported count of model calls. We also include a self-consistency baseline: the joint-input model sampled as many times as our system makes calls, with a majority vote. This separates the benefit of extra compute from the benefit of routing.
Why not just…?
Why not just use contrastive decoding like AVCD? AVCD works at the token level and triggers only when the model is uncertain. It therefore cannot catch fabrications the model asserts with confidence, which are the "confidently wrong" outputs that defeat entropy filters. It also cannot express the idea that audio should outrank video for emotion but not for object presence. The threat is real on object-centric AVHBench items, where AVCD may suffice. Our confidence-stratified analysis will show whether our advantage concentrates on high-confidence errors, as predicted, or disappears.
Why not just caption everything to text and use a text model? AVHBench found that ground-truth captions improve accuracy dramatically. That shows perception is the bottleneck; it does not show that transcripts are enough. Transcripts discard pitch and intensity. The emphasis study shows text alone performing at the majority baseline, and CCFQA reports GPT-4o-mini falling from 96.6% cross-lingual consistency to 62.7% cross-modal consistency. We run a caption-only variant: automatic speech recognition plus sound and frame captions, verified by a text model. We expect it to be competitive on object and spoken-content claims and to fail on prosody, emotion and sarcasm claims. If it does not fail there, our premise about acoustic primacy is wrong for these models.
Why not just query the model once per modality and vote? This is the most serious alternative, because the "more is less" result alone predicts that isolating modalities helps. We therefore make it an ablation, not a strawman. Our claim survives only if routing by claim type beats both uniform voting and global reliability weighting, and does so specifically on items where the channels conflict.
Why not just fine-tune on balanced data? Fine-tuning works in-distribution, as the 83.9% shows. But CMM found that even Gemini 1.5 Pro reaches only 14.5% hallucination resistance on audio-language correlations. Building balanced data, as in MUSIC-AVQA v2.0, requires audio swapping and augmentation that do not scale easily to in-the-wild speech. The transfer comparison against CMM will settle whether fine-tuning beats our approach outside its training distribution.
Proposed Experiments
On AVHBench we report accuracy on the audio-to-video and video-to-audio hallucination tasks and on audio-visual matching. On CMM we report perception accuracy and hallucination resistance, broken down by dominance category. The reference points are Gemini 1.5 Pro's overall hallucination resistance of 58.3% and its 14.5% on audio-language correlations, both reported in The Curse of Multi-Modalities. On MUStARD we report speaker-independent F-score against the 63.1% text-plus-audio result of Towards Multimodal Sarcasm Detection. On the new affective-claim set we report the fraction of unsupported claims kept and the fraction of supported claims wrongly deleted. For multi-turn use, we apply Visual Flow's snowballing protocol to conversations about audio-visual clips; no reported result was found on Lacuna for any audio-aware snowballing baseline.
Three checks decide the outcome.
Does the map recover the human ordering for affective claims? In the estimated map, audio-only verifier accuracy should exceed video-only accuracy for emotion and emphasis claims, even for base models whose joint-input behaviour is visually dominant. If it does not, the human hierarchy does not carry over to these models.
Do the gains concentrate where the channels conflict? Gains over uniform voting should be at least twice as large on conflicting items as on agreeing items. If they are equal, the system is just an expensive ensemble.
Does the method stay ahead at matched cost? At equal model calls, the full system must beat the self-consistency baseline. Replacing the arithmetic arbiter with a free-text language-model arbiter should reduce accuracy on strong base models, reproducing the EmoPrefer trade-off. That result would confirm that the narrow yes/no structure, not the use of multiple agents as such, does the work.
References
- XGC-AVis: Towards Audio-Visual Content Understanding with a Multi-Agent Collaborative System
- Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
- AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
- AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models
- The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio
- Multi-modal Conditional Attention Fusion for Dimensional Emotion Prediction
- The detection of emphatic words using acoustic and lexical features
- EmoPrefer: Can Large Language Models Understand Human Emotion Preferences?
- CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation
- Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering
- Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)