TL;DR
MISA is a novel framework for multimodal sentiment analysis that improves performance by focusing on representation learning rather than just complex fusion mechanisms. By projecting text, audio, and visual data into two distinct subspaces—one capturing shared traits across modalities and another capturing unique, modality-specific quirks—MISA provides a cleaner, disentangled view of the data that makes downstream fusion significantly more effective.
What Problem Does This Paper Address?
Multimodal Sentiment Analysis (MSA) uses a combination of text (spoken words), audio (tone, pitch), and visual (facial expressions) signals to determine a speaker's affective state. The core challenge in this field is the "modality gap." Because these signals come from fundamentally different sources, their data distributions are highly heterogeneous.
Historically, researchers have tried to bridge this gap by building increasingly complex fusion mechanisms (like massive tensor products or intricate attention networks) to force these disparate signals together. The authors of this paper argue that instead of putting all the burden on the fusion layer, we should "clean up" and align the representations of the modalities before they are fused.
Methods
MISA (Modality-Invariant and -Specific Representations) shifts the focus to pre-fusion representation learning. After extracting initial features (using BERT for text and LSTMs for audio/visual), the framework projects each modality into two distinct subspaces using feed-forward neural networks:
- Modality-Invariant Subspace: This space captures the underlying commonalities shared across all modalities (e.g., the overall intent of the speaker). To force the modalities to align here, the authors use Central Moment Discrepancy (CMD) loss, which minimizes the distributional gap between text, audio, and video without the instability of adversarial training.
- Modality-Specific Subspace: This space captures the unique, idiosyncratic features of a single modality (e.g., a specific sarcastic tone or a subtle facial twitch). An orthogonality (difference) loss is applied to ensure these features don't overlap with the invariant space or with the specific spaces of other modalities.
To prevent the network from learning trivial or empty representations, a reconstruction loss ensures that the original input can be rebuilt from these two subspaces. Finally, all six resulting vectors (two per modality) are concatenated and passed through a standard Transformer-based self-attention layer to make the final sentiment prediction.
Datasets
The authors evaluated MISA on three benchmark datasets:
- CMU-MOSI: A popular dataset of 93 YouTube monologues annotated for continuous sentiment intensity (regression and classification).
- CMU-MOSEI: A larger, more diverse improvement over MOSI, containing over 23,000 annotated video segments from 1,000 distinct speakers.
- UR_FUNNY: A recently proposed dataset for Multimodal Humor Detection featuring punchlines from TED talks (binary classification).
Key Insights and Ideas
- Pre-fusion representation is a game-changer: By disentangling the shared and private information of each modality, MISA allows a relatively simple Transformer fusion layer to outperform highly complex, state-of-the-art fusion networks.
- The "Best of Both Worlds" approach: Ablation studies showed that using only a shared invariant space is too restrictive, as not all modalities share the exact same polarity stimulus at all times. Retaining modality-specific spaces preserves vital, quirky affective cues that boost accuracy.
- Attention reveals alignment: Visualizing the self-attention weights in the fusion layer showed that the invariant representations contribute equally across all three modalities—proving the CMD loss successfully aligned them. Meanwhile, specific representations contribute variably depending on the dataset.
Key Results
MISA achieved state-of-the-art results across the board:
- MOSI: Achieved a Mean Absolute Error (MAE) of 0.783 and binary accuracy of 81.8% / 83.4%, outperforming the previous SOTA model (ICCN).
- MOSEI: Achieved an MAE of 0.555 and binary accuracy of 83.6% / 85.5%.
- UR_FUNNY: Hit 70.61% accuracy, crushing the previous SOTA (C-MFN at 65.23%). Impressively, MISA achieved this using only the target utterance, beating baseline models that required extra data from surrounding contextual utterances.
Limitations and Future Work
Limitations: The model's performance is heavily dominated by the text modality. Ablation studies showed that removing text caused a massive performance drop (e.g., MOSI accuracy plummeted from 81.8% to 55.5%), whereas removing audio or visual data caused much smaller dips. The authors attribute this to the high quality of manual transcripts and the massive expressive power of pre-trained BERT compared to raw, unfiltered audio/visual signals.
Future Work: The authors plan to apply the MISA framework to other dimensions of affect, such as discrete emotion recognition. They also suggest combining MISA's representation learning phase with other advanced fusion schemes, and exploring alternative mathematical metrics for the similarity and difference losses.
Summary updated