The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition

Shahin Amiriparian, Lukas Christ, Alexander Kathan, Maurice Gerczuk, Niklas Müller, Steffen Klug, Lukas Stappen, Andreas König, Erik Cambria, Björn W. Schuller, Simone Eulitz

Abstract

The 5th Multimodal Sentiment Analysis Challenge (MuSe 2024) advances the field of affective computing by introducing two complex "in-the-wild" tasks: the prediction of 16 social perception traits of CEOs during investor pitches and the detection of spontaneous humor across different languages and cultures. By utilizing a robust baseline architecture combining state-of-the-art audio, visual, and textual Transformers with gated recurrent units (GRUs), this research demonstrates that multimodal fusion is essential for capturing the intricate nuances of human social signaling. This paper outlines the datasets, methodology, and key findings from the 2024 challenge baseline.

1. Introduction

While traditional sentiment analysis has primarily focused on binary or categorical polarities (e.g., positive vs. negative), human communication is significantly more layered. Social interactions involve subtle cues that signal personality, authority, and intent. MuSe 2024 addresses two specific gaps in current AI research: the dynamic perception of social traits in professional high-stakes environments and the cross-cultural generalization of spontaneous humor. By moving away from staged datasets and static questionnaires, the challenge provides a more ecologically valid framework for evaluating multimodal models.

2. The MuSe-Perception Challenge

The MuSe-Perception sub-challenge utilizes the LMU-ELP (Social Perception) dataset. This novel dataset features 177 CEOs of US-listed firms during "roadshow" presentations. Unlike typical emotion datasets, LMU-ELP focuses on 16-dimensional Likert scale labels, including traits such as arrogance, sincerity, leadership, and likability. These labels were derived from over 4,000 annotators, providing a rich ground truth for how professional figures are perceived by an audience.

The traits are conceptually grouped into the "Big Two" of social perception:

  • Agency: Traits related to competence, dominance, and aggressiveness.
  • Communality: Traits related to friendliness, warmth, and trustworthiness.

3. The MuSe-Humor Challenge

Humor is notoriously difficult for artificial intelligence due to its dependence on cultural context, timing, and delivery. The Passau-SFCH (Humor) dataset addresses this by focusing on spontaneous humor in football coach press conferences.

A unique aspect of this challenge is its cross-lingual and cross-cultural requirement. The training set consists of 10 German Bundesliga coaches speaking German, while the test set features 6 English Premier League coaches of various nationalities speaking English. This setup tests whether models can identify the universal "signals" of humor—such as prosody or facial expressions—independent of the specific language spoken.

4. Methodology and Baseline Architecture

The authors established a baseline using a Gated Recurrent Unit (GRU)-RNN to process sequential multimodal data. The architecture relies on a sophisticated feature extraction pipeline across three modalities:

4.1 Audio Features

The baseline compares traditional "handcrafted" acoustic features (eGeMAPS) against deep representations. These include DeepSpectrum (CNN-based spectrogram features) and Wav2Vec2.0, which was fine-tuned for emotion recognition to capture subtle vocal inflections.

4.2 Visual Features

Video processing begins with isolating faces using MTCNN. From these, the authors extract Facial Action Units (FAUs), FaceNet512 embeddings for identity-linked expressions, and features from a Vision Transformer (ViT-FER) specifically trained for facial emotion recognition.

4.3 Textual Features

For the humor task, multilingual BERT (mBERT) was employed to bridge the gap between German and English transcripts, allowing the model to process semantic content across the two languages.

4.4 Fusion Strategy

The final predictions are generated through weighted late fusion. This involves averaging the predictions of the best-performing unimodal models, with weights determined by their performance on the development set.

5. Key Results and Analysis

5.1 Performance Metrics

  • MuSe-Perception: The baseline achieved a mean Pearson’s Correlation Coefficient ($\rho$) of 0.3573 on the test set. The ViT-FER was the strongest single modality ($\rho = 0.2577$), suggesting that visual cues are highly influential in social trait perception.
  • MuSe-Humor: The multimodal fusion approach reached an Area Under the Curve (AUC) of 0.8682. Interestingly, Wav2Vec2.0 (Audio) and ViT-FER (Video) performed similarly on the test set (AUCs of ~0.80), while BERT followed at 0.7581.

5.2 The Generalization Deficit

A critical insight from the humor task was the "generalization deficit" in visual features. While visual models performed well on the development set, their performance dropped significantly when applied to the cross-cultural test set. Conversely, audio features (Wav2Vec2.0) remained stable, suggesting that the acoustic signature of humor may be more universal than facial expressions across the cultures studied.

5.3 Modality Specialization

No single feature set dominated all 16 social traits in the perception task. For instance, visual features were particularly effective at detecting "agentic" traits like aggressiveness but struggled with internal states like confidence. This reinforces the necessity of multimodal systems to capture the full spectrum of human social signaling.

6. Limitations and Future Work

Despite the robust baseline, several challenges remain:

  • Subject Diversity: The humor dataset is currently limited to male football coaches, which may not reflect humor styles across different genders or professional domains.
  • LLM Integration: While BERT was used for the baseline, the authors suggest that Large Language Models (LLMs) like LLaMA or Mistral could offer superior semantic understanding of complex humor.
  • Class Imbalance: In the humor dataset, humorous segments are sparse (4-6% of the total data), requiring advanced techniques to handle label sparsity.

7. Conclusion

The MuSe 2024 challenge highlights the complexity of moving sentiment analysis into the realm of social perception and spontaneous, cross-cultural humor. The baseline results demonstrate that while deep learning models are becoming proficient at identifying these signals, the integration of multiple modalities remains the most effective strategy for navigating the nuances of human communication.

Summary updated

BibTeX
@inproceedings{amiriparian2024muse,
  author = {Shahin Amiriparian and Lukas Christ and Alexander Kathan and Maurice Gerczuk and Niklas Müller and Steffen Klug and Lukas Stappen and Andreas König and Erik Cambria and Björn W. Schuller and Simone Eulitz},
  title = {{The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition}},
  booktitle = {MuSe@ACM Multimedia},
  year = {2024},
  doi = {10.1145/3689062.3689088},
}
Figure