TL;DR
Large Language Models (LLMs) used for multimodal fusion often fall into a "language shortcut," prioritizing text transcripts over non-verbal cues like facial expressions or tone of voice. This paper introduces MR-Bench to diagnose this bias and proposes CoVeR (Counterfactual-View Reliability learning), a training strategy that forces models to act as unimodal judges, effectively suppressing the text-bias and achieving state-of-the-art results across five sentiment benchmarks.
What Problem Does This Paper Address?
In the world of multimodal learning—where models process text, audio, and video simultaneously—researchers are increasingly using LLMs as "fusion engines." These models treat different modalities as token blocks. However, because LLMs are pre-trained primarily on text, they develop a massive "language prior."
The authors identify a critical failure: when the text contradicts non-verbal evidence (e.g., a speaker saying "I'm so happy" with a sarcastic, miserable face), the model often ignores the visual/audio evidence and follows the text. This "language shortcut" is particularly problematic in Incomplete Multimodal Learning (IML), where some data streams might be missing or corrupted. The paper asks a fundamental question: "Is a face or a voice still heard once the words are known?"
Methods
The researchers tackle this through a two-pronged approach: diagnosis and correction.
- Diagnosis (MR-Bench): They created a benchmark specifically designed to test modality conflict. They identified "Language Shortcut" (LS) cases where the text is misleading and "Non-verbal Shortcut" (NS) cases where the non-verbal cues are the ones that mislead.
- Correction (CoVeR): They proposed Counterfactual-View Reliability learning (CoVeR). Instead of just training the model on the full multimodal input, CoVeR:
- Trains the model to generate "counterfactual views" (e.g., what would the prediction be if I only saw the video?).
- Treats these views as unimodal judges.
- Gated Supervision: If the unimodal judges disagree (e.g., the text judge says "positive" but the video judge says "negative"), the model withholds the fused task supervision. This prevents the model from simply learning to map the dominant text modality to the label while ignoring the others.
Crucially, CoVeR changes the objective during training but keeps the architecture the same, meaning there is no extra computational cost during inference.
Datasets
The study utilizes several established multimodal sentiment analysis benchmarks:
- CMU-MOSI and CMU-MOSEI (English)
- CH-SIMS and CH-SIMSv2 (Chinese)
- IEMOCAP (Emotion recognition)
New Benchmark: MR-Bench * MR-Bench-A: An automated "typing" of every test sample in the above datasets based on unimodal expert judgments. * MR-Bench-H: A "Hard" set of 900 human-validated clips (620 LS, 280 NS) from seven different sources, specifically curated to contain clear modality conflicts like sarcasm or irony.
Key Insights and Ideas
- Dominant-Modality Collapse: By analyzing the internal states of the LLM decoder, the authors found that as the data moves through the layers, the fused representation aligns increasingly with the text view. By the final layers, the non-verbal evidence is essentially "overwritten."
- The Transcript Penalty: In zero-shot Multimodal LLMs (like InternVL or Qwen), adding a transcript to a muted video actually lowers accuracy by 46.1–53.1 points on conflict cases. The model was right when it was "blind" to the words but became wrong once it could read them.
- Inversion Paradox: Simply inverting the answer of a text-only model on LS clips yields 94.4% accuracy. This proves that high accuracy on these benchmarks doesn't necessarily mean the model is "reasoning" multimodally; it might just be lucky or biased.
Key Results
- State-of-the-Art: CoVeR achieves SOTA results on all five benchmarks, outperforming nine other IML methods.
- Robustness: On the difficult MR-Bench-H, CoVeR reaches 61.7% LS accuracy and 93.6% NS accuracy, significantly higher than standard LLM-Fusion (58.8% and 93.1% respectively).
- Reduced Overriding: Among LS clips that the model could solve using only audio/video, adding text caused errors in 30.1% of cases with CoVeR, compared to 32.2% with standard fusion.
- Zero-Shot Failure: Existing state-of-the-art MLLMs (like Qwen2.5-Omni) solve at most 32.7% of the LS clips, performing far below random chance on these conflict cases.
Limitations and Future Work
- Training Overhead: While inference is "free," training CoVeR takes significantly longer (about 5.7x) because it requires processing multiple counterfactual views for every sample.
- Label Dependency: CoVeR currently relies on having unimodal labels (human annotations or expert predictions for each modality separately), which may not always be available for new datasets.
- Dominant Modality Assumption: The method assumes a known dominant modality (text, in this case). Future work could explore how to handle datasets where the dominant modality shifts dynamically.
- Shortcut Detection: MR-Bench-H currently only contains shortcuts that are detectable by existing models; there may be more subtle "human-only" shortcuts not yet captured.
Summary updated