Abstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Introduction
Multimodal Sentiment Analysis (MSA) aims to determine a speaker's affective state by integrating text (spoken words), audio (tone, pitch), and visual (facial expressions) signals. A primary obstacle in this domain is the "modality gap"—the inherent heterogeneity in data distributions across different input sources. Traditional research has focused on bridging this gap through increasingly complex fusion architectures, such as high-order tensor products or multi-head attention mechanisms.
The MISA framework proposes a paradigm shift: rather than relying solely on the fusion layer to resolve modality differences, we should first learn disentangled and aligned representations. By structuring the data into shared and private components before fusion, the subsequent integration process becomes significantly more effective.
The MISA Framework
MISA (Modality-Invariant and -Specific Representations) focuses on pre-fusion representation learning. The architecture consists of four main components: feature extraction, modality representation learning, a decoder for reconstruction, and a fusion layer for prediction.
Subspace Projection
After extracting initial features using BERT for text and LSTMs for audio/visual data, MISA projects each modality into two distinct subspaces:
- Modality-Invariant Subspace: This space captures shared information across modalities. To align these distributions, the framework employs Central Moment Discrepancy (CMD) loss, which minimizes the distance between the representations of different modalities without the instability of adversarial training.
- Modality-Specific Subspace: This space captures unique, idiosyncratic features (e.g., a specific sarcastic tone or a subtle facial expression). An orthogonality loss ensures these private features do not overlap with the shared invariant space or with the specific spaces of other modalities.
To ensure the learned representations are meaningful and not trivial, a reconstruction loss is used to rebuild the original inputs from the combined invariant and specific vectors.
Experimental Evaluation
The authors evaluated MISA across three major benchmarks:
- CMU-MOSI: 93 YouTube monologues annotated for sentiment intensity.
- CMU-MOSEI: A larger dataset with over 23,000 video segments.
- UR_FUNNY: A dataset for Multimodal Humor Detection from TED talks.
Key Results
MISA achieved state-of-the-art (SOTA) performance across all datasets. On MOSI, it reached a binary accuracy of 83.4%, surpassing the previous ICCN baseline. On UR_FUNNY, MISA achieved 70.61% accuracy, significantly outperforming the previous SOTA (C-MFN at 65.23%) while using only the target utterance without additional context.
Analysis and Insights
Ablation studies confirm that both invariant and specific representations are crucial. While the invariant space aligns common intent, the specific spaces preserve vital cues that are unique to a single modality.
Visualizing the learned spaces via t-SNE demonstrates the impact of the disentanglement strategy. As shown above, when the difference loss is applied (right), the modality-specific representations become more distinct and structured, which aids the fusion layer in identifying relevant features for sentiment prediction.
Conclusion
MISA demonstrates that effective representation learning can simplify the multimodal fusion problem. While the model's performance remains heavily influenced by the text modality—likely due to the high quality of manual transcripts and the power of pre-trained BERT—the inclusion of aligned audio and visual features provides a measurable boost. Future work will explore applying MISA to discrete emotion recognition and integrating it with more advanced fusion schemes.
Summary updated