Trimodal Attention Module for Multimodal Sentiment Analysis (Student Abstract)

Anirudh Bindiganavale Harish, Fatiha SadatAAAI 2020 · DOI

Introduction

Sentiment analysis is the computational study of people's opinions, sentiments, and emotions. While analyzing text is a well-established field, human sentiment is inherently multimodal, expressed through a combination of spoken words, vocal tone, and facial expressions. Integrating these diverse signals—a process known as fusion—remains a significant challenge in AI.

Existing models typically rely on either Feature-level (Early) Fusion, which mixes raw signals but risks introducing noise, or Decision-level (Late) Fusion, which aggregates independent predictions but misses the nuanced interactions between modalities (e.g., how a specific tone of voice might signal sarcasm in a sentence). This paper proposes a Trimodal Attention Module (TAM) that bridges this gap by employing a structured, two-stage fusion process.

The Trimodal Attention Module (TAM) Architecture

The proposed network architecture is designed to capture both the unique characteristics of individual modalities and their cross-modal dependencies. The process follows a hierarchical pipeline:

  1. Feature Extraction: Raw signals from text, audio, and video are processed into feature vectors using specialized tools like LSTMs, CNNs, and openSmile.
  2. Stage 1: Feature-Level Fusion via TAM: Each modality is passed through dense layers before entering the Tri-modal Attention Module. This module utilizes Multi-Head Attention (MHA) units that share weights across all three modalities. This weight-sharing strategy significantly reduces the computational footprint while allowing the model to learn a unified representation of sentiment across different data types. Furthermore, a "look-ahead mask" is implemented to ensure the model only considers previous context, mimicking real-time human processing.
  3. Stage 2: Decision-Level Fusion: After the attention stage, the model generates three intermediate classifications. These are then combined using a weighted sum to produce the final sentiment prediction.
The architecture of the proposed network, illustrating the flow from modality-specific features through the Tri-modal Attention Module to the final weighted classification.

Experimental Setup and Results

The model was evaluated on the CMU-MOSI (Multimodal Opinion Sentiment Intensity) dataset, which consists of 2,199 utterances from 93 videos. For these experiments, the original sentiment intensity scores were simplified into a binary classification (positive vs. negative).

Performance Comparison

The TAM-based network was compared against several state-of-the-art (SOTA) models, including Tensor Fusion Networks (TFN) and Memory Fusion Networks (MFN). As shown in the table below, the proposed model achieved a state-of-the-art F1-score of 83.8%, outperforming the previous best (VAE+bc-LSTM) by over 3 percentage points. While the bc-LSTM model maintained a slight lead in raw accuracy, the TAM model's superior F1-score indicates a more robust balance between precision and recall.

Comparison of the proposed network against state-of-the-art methods on the CMU-MOSI dataset.

The Dominance of Text

Through an exhaustive search for optimal fusion coefficients, the researchers determined that the most effective weights were $\alpha=0.5$ (Text), $\beta=0.3$ (Audio), and $\gamma=0.2$ (Video). This finding reinforces the idea that while audio and video provide critical context—especially in cases where a speaker's face is "stoic" or their voice is ambiguous—the transcript remains the primary signal for sentiment analysis.

Conclusion and Future Work

The Trimodal Attention Module demonstrates that combining shared-weight attention with a two-stage fusion strategy can significantly enhance the robustness of multimodal sentiment analysis. By effectively balancing the strengths of text, audio, and video, the model achieves superior performance on benchmark datasets.

Future research will focus on evolving the manual weight tuning into an end-to-end trainable process, allowing the model to learn the optimal $\alpha, \beta, \gamma$ coefficients automatically. Additionally, the authors plan to expand the model's capabilities to multi-task learning for specific emotion identification and test its scalability on larger datasets like MOSEI.

Summary updated

BibTeX
@inproceedings{harish2020trimodal,
  author = {Anirudh Bindiganavale Harish and Fatiha Sadat},
  title = {{Trimodal Attention Module for Multimodal Sentiment Analysis (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7173},
}
Figure