TL;DR
This paper introduces HCMNet, a novel approach for multimodal sentiment analysis that models both fine-grained atomic-level correlations and coarser composition-level relationships between text, audio, and video. By incorporating semantic distribution constraints at both levels, HCMNet effectively addresses inconsistencies across modalities, leading to improved sentiment prediction.
What Problem Does This Paper Address?
Multimodal Sentiment Analysis (MSA) aims to understand human emotions expressed through various data types like text, audio, and video. While existing methods often focus on learning strong correlations at the atomic level (e.g., individual words or video frames), they tend to overlook more complex, weaker correlations that emerge at a higher, compositional level (e.g., relationships between phrases or sequences of frames). Furthermore, these methods often struggle with semantic incongruities between different modalities, where the same sentiment might be expressed differently across text, audio, or video. This paper tackles these limitations by proposing a hierarchical approach that models both atomic and compositional correlations while explicitly addressing and mitigating semantic distribution inconsistencies.
Methods
The proposed Hierarchical Correlation Modeling Network (HCMNet) has three main components:
- Feature Representation: Standard pre-trained models (RoBERTa for text, EfficientNet for video, and librosa for audio) are used to extract initial unimodal features. These are then further processed by modality-specific heads and attention/fusion modules to capture long-term dependencies within each modality.
- Atomic-level Correlation Modeling (ACM): This component focuses on learning correlations between individual elements (tokens, frames, segments) across modalities. It uses a dynamic attention reasoning method to align features into a common space and compute similarities. Crucially, it introduces an Atomic-level Contrastive Learning (ACL) loss to constrain semantic distributions, pushing samples of the same emotion closer and different emotions further apart in the feature space, thus mitigating atomic-level inconsistencies.
- Composition-level Correlation Modeling (CCM): This part models higher-level relationships. It constructs uni-modal graphs (text, video, audio) based on semantic dependencies (e.g., dependency relations for text, cosine similarity for video utterances) and then uses Graph Convolutional Networks (GCNs) to learn composition-level features. To address composition-level inconsistencies, a Semantic Optimal Transport (SOT) module is introduced. This module uses optimal transport to learn alignment flows between graph representations of different modalities, effectively constraining their composition-level semantic distributions.
The final prediction is made by an emotion aggregator that combines the atomic and composition-level features.
Datasets
The paper evaluates HCMNet on three widely-used multimodal sentiment analysis benchmark datasets:
- CMU-MOSEI: A large-scale dataset with text, audio, and visual modalities.
- IEMOCAP: A dataset focusing on dyadic emotional interactions.
- MELD: A multimodal multi-party dataset for emotion recognition in conversations.
The paper provides details on data splitting in Table 1. No new datasets were introduced.
Key Insights and Ideas
- Hierarchy Matters: Sentiment is not just about individual elements but also about how they combine. Modeling both atomic and compositional correlations is crucial for robust MSA.
- Inconsistency is a Bottleneck: Semantic differences across modalities (e.g., text saying "happy" while audio sounds sad) significantly hinder performance. Explicitly addressing these inconsistencies is key.
- Contrastive Learning for Atomic Consistency: Using contrastive loss at the atomic level effectively pulls similar emotional features together and pushes dissimilar ones apart, improving feature discriminability.
- Optimal Transport for Compositional Alignment: Optimal transport provides a principled way to align higher-level semantic structures (graphs) across modalities, ensuring better compositional congruity.
- Graph Structures Capture Richer Semantics: Using dependency graphs for text and similarity-based graphs for video/audio allows GCNs to capture complex inter-utterance and inter-token relationships.
Key Results
- HCMNet achieves state-of-the-art performance on IEMOCAP and MELD datasets, outperforming strong baselines like MultiEMO by significant margins (e.g., 2.44% and 3.30% overall Weighted-F1 improvement).
- Ablation studies confirm the effectiveness of both the Atomic-level Correlation Modeling (ACM) and Composition-level Correlation Modeling (CCM) components. The ACM-only model performs well, and the CCM component further boosts performance.
- Removing Atomic-level Contrastive Learning (ACL) or the Composition-level Distribution Constraint (CDC) leads to performance degradation, highlighting the importance of both distribution constraint mechanisms.
- Qualitative analysis using t-SNE visualizations shows that HCMNet with CDC can better align multimodal features with emotional semantic relationships, leading to more distinct clusters for different emotion categories.
- A case study demonstrates HCMNet's ability to provide accurate predictions by leveraging both atomic and compositional information, outperforming previous methods.
Limitations and Future Work
- CMU-MOSEI Performance: While HCMNet performs well, the paper notes that its accuracy and F1-score improvement on CMU-MOSEI is "mediocre," suggesting potential for further optimization on this dataset.
- Dataset Characteristics: The paper observes that IEMOCAP, collected from well-scripted utterances, yields higher F1 scores than MELD, implying that dataset quality and script clarity can influence performance.
- Future Work: The authors suggest incorporating contextual knowledge from large vision-language models to further enhance the learning of emotional correlations.
Summary updated