Sentiment Analysis and Topic Recognition in Video Transcriptions

Lukas Stappen, Alice Baird, Erik Cambria, Björn W. SchullerIEEE Intell. Syst 2021 · DOI · OpenReview

Abstract

Multimodal Sentiment Analysis (MSA) typically relies on data-intensive deep learning models to process visual, audio, and textual data. This research demonstrates that integrating high-level, knowledge-based features from video transcriptions can significantly enhance sentiment and topic recognition. By utilizing SenticNet to bridge the gap between raw text and symbolic meaning, the authors achieved superior performance over complex neural baselines while maintaining low computational overhead.

Introduction

The field of Multimodal Sentiment Analysis (MSA) seeks to decode human emotions and topics by synthesizing information from three primary channels: the visual (what we see), the acoustic (what we hear), and the linguistic (what is said). While deep learning has pushed the boundaries of what is possible in MSA, many current state-of-the-art models are "black boxes" that demand massive datasets and significant computational power.

Furthermore, a critical challenge in MSA is the varying utility of different modalities. While audio and visual channels are highly effective at detecting the intensity of an emotion (arousal), the textual modality—the actual words spoken—is often the most vital for understanding the specific context and the direction of the sentiment (valence). This paper investigates whether "commonsense" knowledge bases can be used to extract more meaningful features from video transcriptions than standard word embeddings.

Methodology: From Speech to Symbolic Meaning

The researchers developed a multi-stage pipeline designed to move from raw video data to structured semantic predictions.

1. Transcription

The first step involves converting spoken words into a machine-readable format. The authors utilized Amazon Transcribe, incorporating a custom dictionary of automotive terms to ensure accuracy within the specific domain of car reviews.

The transcription pipeline converts raw audio into text using specialized tools like Amazon Transcribe.

2. Feature Extraction via SenticNet

Rather than relying on raw word embeddings, the study leverages SenticNet (versions 5 and 6). SenticNet is a knowledge base that maps natural language concepts to a structured framework of meaning:

  • Sentics: These represent core emotional dimensions such as pleasantness, attention, sensitivity, and aptitude.
  • Moodtags: These provide specific emotional labels, such as "joy" or "interest."
  • Semantics: These are concept clusters that provide broader context for the spoken segments.

This approach is grounded in the "Hourglass of Emotions," a biologically inspired framework that provides a hierarchical structure to human communication, which raw neural networks often lack.

3. Modeling Approaches

The authors compared two primary modeling strategies to process these SenticNet features:

  • Support Vector Machines (SVM): This approach used "n-hot" encoded vectors of the SenticNet concepts, providing a transparent and efficient classification method.
  • Neural Networks: This involved fine-tuning semantic concepts into domain-specific embeddings to capture more nuanced relationships.

Experimental Setup and Datasets

The study utilized the MuSe-CAR dataset, a large-scale collection of multimodal YouTube car reviews. The focus was specifically on the MuSe-Topic subchallenge, which required the classification of ten distinct topic classes (e.g., "Interior," "Performance," "Handling") and emotion labels across three levels (low, medium, high) for both valence and arousal. The dataset comprises over 28,000 transcribed sentences, providing a robust testing ground for the proposed methods.

Key Insights and Results

The research yielded several significant findings regarding the power of symbolic knowledge in NLP:

  • Superior Topic Recognition: The best-performing model—an SVM using n-hot encoding from SenticNet 5—achieved a combined challenge metric of 66.16% on the test set. This result significantly outperformed the LSTM baseline (36.20%) and even a complex Multimodal Transformer (52.98%).
  • Enhanced Sentiment Accuracy: The SenticNet-based approach improved the linguistic baseline for valence prediction by nearly 3% absolute.
  • Efficiency vs. Scale: While the massive ALBERT transformer model maintained the highest raw topic accuracy (76.78%), the SenticNet/SVM approach provided a highly competitive alternative that requires only a fraction of the computational resources.
  • The Arousal Gap: The study confirmed that while text is excellent for predicting what is being discussed and whether the speaker likes it (valence), it struggles to predict how excited they are (arousal). Arousal remains better captured by vocal tone and facial expressions.

Conclusion and Future Directions

This research highlights that high-level contextual features can provide a "top-down" understanding of the world that rivals or exceeds the performance of much larger, data-hungry models. By grounding NLP in commonsense knowledge, models gain a level of interpretability and efficiency that is often missing in pure deep learning approaches.

Future work will focus on fusing these high-level linguistic features with audio and visual data—such as vocal pitch and facial muscle movements—to create a truly robust multimodal system. Additionally, the authors aim to explore unsupervised methods for clustering video content based on these "commonsense" semantic features, potentially reducing the need for manual labeling in large datasets.

Summary updated

Abstract

Nowadays, videos are an integral modality for information sharing on the World Wide Web. However, systems able to automatically understand the content and sentiment of a video are still in their infancy. Linguistic information transported in spoken parts of a video is known to convey valuable properties in regards to context and emotions. In this article, we explore a lexical knowledge-based extraction approach to obtain such understanding from the video transcriptions of a large-scale multimodal dataset (MuSe-CAR). To this end, we use SenticNet to extract natural language concepts and fine-tune several feature types on a subset of MuSe-CAR. With these features, we explore the content of a video as well as learning to predict emotional valence, arousal, and speaker topic classes. Our best model improves the linguistic baseline from the MuSe-Topic 2020 subchallenge by almost 3% (absolute) for the prediction of valence on the predefined challenge metric and outperforms a variety of baseline systems that require much higher computational power than the one proposed herein.
BibTeX
@article{stappen2021sentiment,
  author = {Lukas Stappen and Alice Baird and Erik Cambria and Björn W. Schuller},
  title = {{Sentiment Analysis and Topic Recognition in Video Transcriptions}},
  journal = {IEEE Intell. Syst.},
  year = {2021},
  doi = {10.1109/mis.2021.3062200},
}
Figure