Abstract
Multimodal Sentiment Analysis (MSA) typically relies on data-intensive deep learning models to process visual, audio, and textual data. This research demonstrates that integrating high-level, knowledge-based features from video transcriptions can significantly enhance sentiment and topic recognition. By utilizing SenticNet to bridge the gap between raw text and symbolic meaning, the authors achieved superior performance over complex neural baselines while maintaining low computational overhead.
Introduction
The field of Multimodal Sentiment Analysis (MSA) seeks to decode human emotions and topics by synthesizing information from three primary channels: the visual (what we see), the acoustic (what we hear), and the linguistic (what is said). While deep learning has pushed the boundaries of what is possible in MSA, many current state-of-the-art models are "black boxes" that demand massive datasets and significant computational power.
Furthermore, a critical challenge in MSA is the varying utility of different modalities. While audio and visual channels are highly effective at detecting the intensity of an emotion (arousal), the textual modality—the actual words spoken—is often the most vital for understanding the specific context and the direction of the sentiment (valence). This paper investigates whether "commonsense" knowledge bases can be used to extract more meaningful features from video transcriptions than standard word embeddings.
Methodology: From Speech to Symbolic Meaning
The researchers developed a multi-stage pipeline designed to move from raw video data to structured semantic predictions.
1. Transcription
The first step involves converting spoken words into a machine-readable format. The authors utilized Amazon Transcribe, incorporating a custom dictionary of automotive terms to ensure accuracy within the specific domain of car reviews.
2. Feature Extraction via SenticNet
Rather than relying on raw word embeddings, the study leverages SenticNet (versions 5 and 6). SenticNet is a knowledge base that maps natural language concepts to a structured framework of meaning:
- Sentics: These represent core emotional dimensions such as pleasantness, attention, sensitivity, and aptitude.
- Moodtags: These provide specific emotional labels, such as "joy" or "interest."
- Semantics: These are concept clusters that provide broader context for the spoken segments.
This approach is grounded in the "Hourglass of Emotions," a biologically inspired framework that provides a hierarchical structure to human communication, which raw neural networks often lack.
3. Modeling Approaches
The authors compared two primary modeling strategies to process these SenticNet features:
- Support Vector Machines (SVM): This approach used "n-hot" encoded vectors of the SenticNet concepts, providing a transparent and efficient classification method.
- Neural Networks: This involved fine-tuning semantic concepts into domain-specific embeddings to capture more nuanced relationships.
Experimental Setup and Datasets
The study utilized the MuSe-CAR dataset, a large-scale collection of multimodal YouTube car reviews. The focus was specifically on the MuSe-Topic subchallenge, which required the classification of ten distinct topic classes (e.g., "Interior," "Performance," "Handling") and emotion labels across three levels (low, medium, high) for both valence and arousal. The dataset comprises over 28,000 transcribed sentences, providing a robust testing ground for the proposed methods.
Key Insights and Results
The research yielded several significant findings regarding the power of symbolic knowledge in NLP:
- Superior Topic Recognition: The best-performing model—an SVM using n-hot encoding from SenticNet 5—achieved a combined challenge metric of 66.16% on the test set. This result significantly outperformed the LSTM baseline (36.20%) and even a complex Multimodal Transformer (52.98%).
- Enhanced Sentiment Accuracy: The SenticNet-based approach improved the linguistic baseline for valence prediction by nearly 3% absolute.
- Efficiency vs. Scale: While the massive ALBERT transformer model maintained the highest raw topic accuracy (76.78%), the SenticNet/SVM approach provided a highly competitive alternative that requires only a fraction of the computational resources.
- The Arousal Gap: The study confirmed that while text is excellent for predicting what is being discussed and whether the speaker likes it (valence), it struggles to predict how excited they are (arousal). Arousal remains better captured by vocal tone and facial expressions.
Conclusion and Future Directions
This research highlights that high-level contextual features can provide a "top-down" understanding of the world that rivals or exceeds the performance of much larger, data-hungry models. By grounding NLP in commonsense knowledge, models gain a level of interpretability and efficiency that is often missing in pure deep learning approaches.
Future work will focus on fusing these high-level linguistic features with audio and visual data—such as vocal pitch and facial muscle movements—to create a truly robust multimodal system. Additionally, the authors aim to explore unsupervised methods for clustering video content based on these "commonsense" semantic features, potentially reducing the need for manual labeling in large datasets.
Summary updated