Abstract
Multi-modal sentiment analysis (MSA) traditionally operates on the assumption that text, audio, and visual modalities provide complementary information. However, human emotional expression is frequently characterized by semantic inconsistency, such as the use of sarcasm or the masking of distress with neutral language. This paper introduces DiffEmo, a benchmark designed to evaluate model robustness in the face of such modality conflicts. Our evaluation of eleven traditional fusion architectures and state-of-the-art Multimodal Large Language Models (MLLMs) reveals a significant performance degradation when modalities clash, highlighting a critical "text bias" and the limitations of current reasoning capabilities in MLLMs.
1. Introduction
The field of Multi-modal Sentiment Analysis (MSA) has largely focused on the "fusion" problem: how to effectively integrate signals from diverse sources—text, voice, and facial expression—to achieve a more holistic understanding of human emotion. Most existing frameworks assume that additional modalities provide redundant or additive information that clarifies the underlying sentiment.
In reality, human communication is nuanced and often contradictory. Social norms or personal psychological states can lead individuals to express conflicting signals—for instance, maintaining a calm exterior while speaking with an anxious tone. This research addresses the gap in current MSA benchmarks by specifically investigating semantic inconsistency, questioning whether models can resolve conflicts when different modalities suggest opposing emotional states.
2. The DiffEmo Dataset
To quantify the impact of modality conflict, the researchers developed the DiffEmo Dataset, derived from the CH-SIMS v2.0 Chinese multi-modal dataset. A "conflicting sample" is defined as an instance where the sentiment scores between any two modalities differ by more than 1.0 on a scale ranging from -1 (negative) to 1 (positive).
The dataset is categorized into three distinct subsets:
- Aligned Set (173 samples): Instances where all modalities are semantically coherent.
- Conflicting Set (173 samples): Instances featuring severe semantic discrepancies between modalities.
- Mixed Set: A combination of both aligned and conflicting data to simulate real-world conditions.
3. Methodology
The study evaluates two primary categories of models:
3.1 Traditional Deep Learning Architectures
Eleven architectures were tested, representing four major fusion strategies:
- Early Fusion (e.g., EF-LSTM): Features are concatenated at the input level.
- Late Fusion (e.g., LF-DNN): Predictions are made independently for each modality and then aggregated.
- Tensor/Hybrid Fusion (e.g., TFN): Employs complex mathematical operations to model inter-modal interactions.
- Translation/Transformer Fusion (e.g., MulT): Uses attention mechanisms to capture cross-modal dependencies.
3.2 Multimodal Large Language Models (MLLMs)
The researchers evaluated Video-LLaMA (7B version) using In-Context Learning (ICL). The model was provided with a four-shot prompt containing examples of emotional reasoning to determine if it could navigate conflicting signals through natural language inference.
4. Results and Analysis
4.1 The Performance Gap
The most striking finding was the universal performance drop across all models when transitioning from aligned to conflicting data. For example, the EF-LSTM model, which achieved 85.9% accuracy on the Aligned Set, plummeted to 67.6% on the Conflicting Set. This suggests that current fusion strategies are heavily reliant on modal agreement.
4.2 Complexity vs. Robustness
Interestingly, simpler models like EF-LSTM and MLF-DNN (Late Fusion) outperformed many complex counterparts on conflicting data. This implies that sophisticated fusion mechanisms may inadvertently "overfit" to the assumption of modality complementarity, making them less robust when that assumption is violated. However, the Multimodal Transformer (MulT) demonstrated the highest overall stability, suggesting that cross-modal attention is a more resilient way to handle messy, long-range semantics.
4.3 The "Text Bias"
The study identified a significant "text bias" in MSA. Textual data was found to be the modality most likely to conflict with the ground-truth multi-modal label. This aligns with human behavior, where verbal content is often used to mask genuine emotional states that are more accurately reflected in audio or visual cues.
4.4 MLLM Limitations
Despite their supposed reasoning capabilities, MLLMs like Video-LLaMA struggled significantly. When tasked with identifying whether a video contained inconsistent signals, the model achieved a "True" detection rate of only 53.68%—barely better than random chance. Furthermore, MLLMs showed high sensitivity to the examples provided in their prompts; if the prompt only contained "aligned" examples, the model's performance on conflicting test data deteriorated further.
5. Discussion and Future Work
The findings suggest that MSA needs to move beyond simple pattern matching toward "emotional reasoning." Current models often default to a single modality (usually text) or hallucinate a consensus when faced with conflict.
The authors acknowledge limitations, including the use of the smaller 7B Video-LLaMA model due to hardware constraints and the simplification of the task to single-speaker clips. Future research should focus on the development of "video emotion entailment" datasets. Such datasets would train models to recognize the logical relationship—or lack thereof—between what is said and what is shown, treating inconsistency as a feature to be analyzed rather than noise to be fused.
6. Conclusion
The DiffEmo benchmark reveals a critical vulnerability in modern sentiment analysis models. As AI systems are increasingly deployed in human-centric environments, the ability to detect and resolve emotional inconsistency is paramount. Moving forward, the field must prioritize architectures that can reason through conflicting signals rather than simply assuming they will always align.
Summary updated