Multimodal Characterization of Emotion within Multimedia Space

Dayo Samuel Banjo, Connice Trimmingham, Niloofar Yousefi, Nitin AgarwalarXiv 2023 · OpenReview

TL;DR

This paper tackles the challenge of accurately understanding emotions in online multimedia content by proposing a multimodal approach that fuses audio and textual information. By leveraging transformer-based models like VisualBERT, they demonstrate that combining these modalities significantly outperforms single-modality analysis, leading to more robust and nuanced emotion characterization.

What Problem Does This Paper Address?

The explosion of social media and online content has made understanding user emotions crucial for effective human-computer interaction. However, emotions are complex and often expressed through multiple channels simultaneously (e.g., spoken words, tone of voice, accompanying text). Previous research has shown that relying on a single modality (like just text or just audio) for emotion detection is often insufficient, leading to limited accuracy and an incomplete picture of the user's emotional state. This paper aims to bridge this gap by exploring how to effectively combine information from different modalities to achieve a more comprehensive and accurate characterization of emotions within multimedia content.

Methods

The core of this research lies in leveraging advanced deep learning architectures, specifically transformer models, for multimodal emotion analysis. The authors employ two main strategies:

  • Unimodal Coherence Analysis: They first analyze the coherence between unimodal emotion detection systems for audio and text. This involves using a pre-trained DistilBERT-base-uncased-emotion model for text and the librosa library for audio feature extraction. The goal here is to understand how well emotions detected from text align with those detected from audio within the same content.
  • Multimodal Single Shot Analysis: For a more integrated approach, they fine-tune the VisualBERT model. VisualBERT is a transformer architecture designed to jointly process vision and language, and in this study, it's adapted to handle text and audio. The input to VisualBERT is a combination of a spectrogram image of the audio and the transcribed text. This allows the model to learn contextual relationships between these modalities in a single pass.

Datasets

The study utilizes two primary datasets:

  • YouTube-1K Dataset: This dataset was generated by querying the YouTube API for video metadata and content. It comprises 1000 video IDs from 8 categories (games, travel, sports, entertainment, science, news, politics), including audio, frames, title, and descriptions. This dataset was used for unimodal emotion analysis.
  • IEMOCAP Dataset: This is a well-established dataset for multimodal emotion assessment. It contains 1440 multimedia files with audio, visual, and text transcriptions from acted, multispeaker scenarios. IEMOCAP is annotated for multiple emotion categories, including neutrality, sadness, anger, surprise, fear, and happiness. This dataset was used for both unimodal and multimodal analysis and comparison.

Key Insights and Ideas

  • Multimodality is Key: The research strongly reinforces the idea that combining multiple modalities (text and audio) leads to significantly better emotion detection performance compared to using any single modality alone.
  • Coherence Varies by Emotion: Not all emotions are equally coherent across modalities. The study found that emotions like happiness, anger, fear, and love showed higher coherence between text and audio signals, suggesting these emotions are more consistently expressed across different communication channels. Sad and surprise emotions, however, showed weaker coherence, indicating they might be more subtly or inconsistently expressed.
  • Transformer Power: The effectiveness of transformer-based models, particularly VisualBERT, is highlighted for its ability to jointly process and align information from different modalities, leading to improved contextual understanding.
  • Beyond Simple Fusion: The study moves beyond simple concatenation of features, using VisualBERT's inherent ability to align textual and visual (in this case, audio spectrograms) elements through its self-attention mechanism.

Key Results

  • Higher Accuracy with Multimodality: The multimodal approach, specifically using VisualBERT, achieved higher accuracy in emotion assessment compared to unimodal (audio-only) analysis on the IEMOCAP dataset. Table 2 shows unimodal achieving 71.6% test accuracy, while the multimodal approach (though not explicitly quantified in the table, implied by the text and figures) demonstrates superior performance.
  • Improved Confusion Matrices: The confusion matrices (Fig. 5) clearly illustrate the advantage of the multimodal approach. The multimodal confusion matrix shows a more concentrated distribution along the diagonal, indicating fewer misclassifications and better discrimination between emotion classes compared to the unimodal matrix.
  • Stronger AUROC: The Area Under the Receiver Operating Characteristic (AUROC) curve for the multimodal assessment was significantly higher (0.993) than for the unimodal assessment (0.922), indicating superior separability of emotion classes.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that while the selected videos showed high coherence for certain emotions, the analysis of sad and surprise emotions revealed weak coherence, suggesting these might be more challenging to detect reliably across modalities.
  • The paper focuses on text and audio. The authors note that incorporating other modalities like video and image could further enhance emotion characterization.
  • Future Work:
    • The authors suggest exploring more advanced multimodal features and architectures, potentially incorporating video and image data alongside text and audio.
  • Future research could also delve deeper into understanding the specific interplay and coherence of different emotion classes across various modalities.
  • The potential for applying these multimodal emotion recognition systems to real-world applications like social media analysis, content recommendation, and user experience improvement is a clear avenue for future exploration.

Summary updated

Abstract

Technological advancement and its omnipresent connection have pushed humans past the boundaries and limitations of a computer screen, physical state, or geographical location. It has provided a depth of avenues that facilitate human-computer interaction that was once inconceivable such as audio and body language detection. Given the complex modularities of emotions, it becomes vital to study human-computer interaction, as it is the commencement of a thorough understanding of the emotional state of users and, in the context of social networks, the producers of multimodal information. This study first acknowledges the accuracy of classification found within multimodal emotion detection systems compared to unimodal solutions. Second, it explores the characterization of multimedia content produced based on their emotions and the coherence of emotion in different modalities by utilizing deep learning models to classify emotion across different modalities.
BibTeX
@misc{banjo2023multimodal,
  author = {Dayo Samuel Banjo and Connice Trimmingham and Niloofar Yousefi and Nitin Agarwal},
  title = {{Multimodal Characterization of Emotion within Multimedia Space}},
  year = {2023},
  eprint = {2311.11892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.11892},
}
Figure