Abstract
In the last decade, video blogs (vlogs) have become an extremely popular method through which people express sentiment. The ubiquitousness of these videos has increased the importance of multimodal fusion models, which incorporate video and audio features with traditional text features for automatic sentiment detection. Multimodal fusion offers a unique opportunity to build models that learn from the full depth of expression available to human viewers. In the detection of sentiment in these videos, acoustic and video features provide clarity to otherwise ambiguous transcripts. In this paper, we present a multimodal fusion model that exclusively uses high-level video and audio features to analyze spoken sentences for sentiment. We discard traditional transcription features in order to minimize human intervention and to maximize the deployability of our model on at-scale real-world data. We select high-level features for our model that have been successful in nonaffect domains in order to test their generalizability in the sentiment detection domain. We train and test our model on the newly released CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset, obtaining an F1 score of 0.8049 on the validation set and an F1 score of 0.6325 on the held-out challenge test set.
Introduction
The proliferation of video blogs (vlogs) has transformed how sentiment is expressed and captured online. While traditional sentiment analysis has focused heavily on text, the multimodal nature of vlogs—incorporating facial expressions, gestures, and vocal tone—presents a richer data source. However, most multimodal models remain tethered to human-generated transcripts, creating a significant bottleneck for real-world, at-scale applications.
In the lead paper for the 2018 Grand Challenge on Human Multimodal Language, "Getting the subtext without the text," Blanchard et al. propose a paradigm shift: a sentiment classification pipeline that completely ignores linguistic content in favor of visual and acoustic "subtext."
The Problem: The Transcription Bottleneck
Traditional multimodal research often treats text as the primary modality, with video and audio acting as supplementary features. This approach relies on manual transcription, which is prohibitively expensive and slow for massive datasets. While Automatic Speech Recognition (ASR) offers an alternative, it often fails to capture the nuances of "in-the-wild" social media content.
Furthermore, text alone is frequently ambiguous. A transcript cannot easily distinguish between a sincere "Great" and a sarcastic one. By focusing on how something is said (prosody) and the speaker's physical delivery (gestures and expressions), the authors aim to build a model that is both scalable and capable of resolving linguistic ambiguity without human intervention.
Methodology: A Bag-of-Features Approach
The researchers implemented a three-level Bag-of-Features (BoF) pipeline, adapting techniques typically reserved for computer vision to the domain of sentiment analysis.
1. Feature Extraction
- Visual Modality: The model utilizes Temporal Robust Features (TRoF). Originally designed for detecting motion-heavy content like violence, TRoF isolates spatiotemporal segments of video to describe movement. This study tests whether these features can generalize to the subtle movements associated with human affect.
- Acoustic Modality: Using the openSMILE toolkit, the team extracted prosodic features including pitch, loudness, and voicing probability. Crucially, the model ignores the phonemes and words, focusing entirely on the non-verbal acoustic signal.
2. Mid-level Representation
To bridge the gap between raw signals and classification, the authors used Gaussian Mixture Models (GMMs) to create "codebooks." These codebooks serve as a vocabulary of visual and acoustic patterns, effectively quantizing the data into a "Bag-of-Visual-Words" and "Bag-of-Audio-Words."
3. Classification and Fusion
Separate Linear Support Vector Machines (SVMs) were trained for each modality. The authors then explored two fusion strategies:
- Score-level fusion: Averaging the confidence scores from both the audio and video models.
- Output-level fusion: Binning and thresholding the raw scores to produce a final sentiment prediction.
Dataset: CMU-MOSEI
The model was evaluated using the CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset. At the time of the study, this was a significant upgrade over previous datasets, containing over 23,500 sentence-level videos (65+ hours) from more than 1,000 diverse YouTube speakers covering a wide array of topics.
Results and Discussion
The experimental results highlight a clear trade-off between scalability and peak accuracy.
- Performance: The output-level fusion model achieved the best results on the CMU-MOSEI validation set with an F1 score of 0.8049 and a Mean Absolute Error (MAE) of 0.7760. On the more rigorous Grand Challenge held-out test set, it reached an F1 score of 0.6325.
- Modality Strength: A key finding was the disparity between modalities. The video-only model (F1: 0.75) significantly outperformed the audio-only model (F1: 0.58). This suggests that while prosody is valuable, visual cues captured by TRoF are more robust indicators of sentiment when linguistic context is absent.
- Generalizability: The success of TRoF features—originally intended for non-affect domains—demonstrates that high-level motion features are surprisingly effective at capturing human sentiment.
Conclusion and Future Work
Blanchard et al. demonstrate that a "text-free" approach is viable for large-scale sentiment detection, prioritizing deployability. While the exclusion of text results in lower accuracy compared to transcript-reliant models, the BoF approach offers a path toward analyzing the vast quantities of unlabelled video data generated daily.
Future work may look to incorporate "noisy" text from ASR to bridge the performance gap or improve the mapping of SVM distances to sentiment intensity. For now, the study serves as a benchmark for how much "subtext" can be extracted when the words themselves are stripped away.
Summary updated