¡Qué maravilla! Multimodal Sarcasm Detection in Spanish: a Dataset and a Baseline

Khalid Alnajjar, Mika HämäläinenNAACL 2021

TL;DR

This paper introduces the first multimodal sarcasm detection dataset for Spanish, combining text, audio, and video from animated TV shows. They demonstrate that incorporating audio and video significantly improves sarcasm detection accuracy over text-only models, with the best multimodal model achieving 93.1% accuracy.

What Problem Does This Paper Address?

Sarcasm, a complex form of figurative language, is notoriously difficult for machines to understand. While previous research has explored sarcasm detection, much of it has focused on English text. This paper addresses the critical need for a Spanish-language resource that leverages multimodal information (text, audio, and video) to capture the nuances of sarcastic expression, which often relies on context, tone, and visual cues. The authors highlight that existing Spanish resources are limited, and multimodal approaches are even scarcer, especially for languages beyond English.

Methods

The authors developed a multimodal sarcasm detection system by combining various machine learning techniques. For text-only analysis, they experimented with a bi-directional LSTM model (using OpenNMT) and a Support Vector Machine (SVM) that incorporated GloVe and ELMo embeddings. To incorporate audio, they extracted 13 features using librosa, including STFT, MFCC, and chroma, and fed these into an SVM model alongside text features. For the full multimodal approach, they extracted visual features from three key frames per utterance using a pre-trained ResNet-152 model and combined these with text and audio features as input to an SVM.

Datasets

The core contribution is a newly constructed multimodal sarcasm dataset for Spanish. This dataset is derived from the MA thesis of the second author and includes annotated text from two South Park episodes (Latin American Spanish) and two Archer episodes (Peninsular Spanish). Crucially, they aligned this existing text data with corresponding video and audio, and also introduced scene annotations to better capture contextual information. They also fixed errors in the original transcriptions. The dataset is available on Zenodo, with an open-access version containing only text and a request-based version including videos.

Key Insights and Ideas

  • Multimodality is Key for Spanish Sarcasm: The results strongly suggest that for Spanish sarcasm, relying solely on text is insufficient. Audio and visual cues provide crucial context that significantly boosts detection accuracy.
  • Context is King (and Queen, and Jack): Sarcasm is highly context-dependent. The paper illustrates how the same utterance can be sarcastic or sincere depending on the speaker, the situation, and even visual elements like an unfashionable neckpiece.
  • Animation Presents Unique Challenges/Opportunities: The use of animated shows allows for visual cues that might be less obvious in live-action, but also means that frame-to-frame changes might be less dynamic, influencing how video is processed.
  • SVMs Outperform LSTMs in this Scenario: The SVM models, particularly when combined with multimodal features, achieved higher accuracy than the LSTM-based neural model, likely due to the SVM's ability to handle high-dimensional data and the use of pre-trained embeddings.

Key Results

The paper presents a clear progression in accuracy as more modalities are incorporated:

  • Text-only: Neural Model: 87.5%, SVM: 89.0%
  • Text + Audio: SVM: 91.9%
  • Text + Audio + Video: SVM: 93.1%

The best-performing model, combining all three modalities, achieved a 4.1% gain in accuracy over the text-based SVM model.

Limitations and Future Work

The authors acknowledge that sarcasm detection remains a challenging task, even with multimodal information. Their error analysis revealed that models can learn to associate certain features with sarcasm (e.g., a recurring visual element) but may overgeneralize, leading to misclassifications. They note that the model learned the data but not necessarily the underlying phenomenon of sarcasm, which requires a deeper "global understanding of the world and the specific context." Future work could explore more sophisticated multimodal fusion techniques and investigate how to better capture the nuanced contextual understanding required for true sarcasm detection.

Summary updated

BibTeX
@inproceedings{alnajjar2021que,
  author = {Khalid Alnajjar and Mika Hämäläinen},
  title = {{¡Qué maravilla! Multimodal Sarcasm Detection in Spanish: a Dataset and a Baseline}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure