On the Complementarity of Images and Text for the Expression of Emotions in Social Media

Anna Khlyzova, Carina Silberer, Roman KlingerACL 2022

TL;DR

This paper explores how images and text complement each other to express emotions on social media. By creating a new annotated dataset of Reddit posts, the researchers found that while text often suffices for emotion detection, images can be crucial for understanding the stimulus of an emotion and sometimes even the emotion itself, especially in nuanced cases.

What Problem Does This Paper Address?

Social media platforms are rich with emotional expression, often conveyed through a combination of text and images. While previous research has focused on analyzing emotions from text or images independently, there's a significant gap in understanding how these two modalities interact and complement each other to convey emotional meaning. This is particularly relevant as visual platforms like TikTok and Instagram gain prominence. The authors aim to quantify this complementarity, investigating whether images add unique emotional information beyond what text alone provides, and how this interplay influences the identification of both the emotion and its underlying cause (stimulus).

Methods

The core of this research lies in creating and analyzing a novel multimodal dataset. The authors developed a detailed annotation scheme for Reddit posts, covering:

  • Emotion Classification: Using the eight basic emotions from Plutchik's wheel (anger, anticipation, joy, sadness, trust, surprise, fear, disgust).
  • Image-Text Relation Classification: Categorizing how images relate to text in conveying emotion, with labels like "complementary" (image essential), "illustrative" (image explains text), "opposite" (contradictory), "decorative" (image for aesthetics), and "emotion communicated with image only."
  • Emotion Stimulus Classification: Identifying the visual element that triggers the emotion, with categories such as person/people, animal, object, food, meme, screenshot/text in image, art/drawing, advertisement, event/situation, and place.

For modeling, they employed standard deep learning techniques:

  • Text-based Model: Fine-tuned a pre-trained RoBERTa model for multi-task learning of emotion, stimulus, and relation.
  • Image-based Model: Utilized a pre-trained ResNet50 model, adding fully connected layers for classification.
  • Multimodal Models: Explored three fusion strategies:
  • Early Fusion: Concatenating features from both modalities before classification.
  • Late Fusion: Combining the output probabilities of unimodal models.
  • Model-based Fusion: Fusing features from the last hidden layers of unimodal models.

Datasets

The primary dataset introduced in this paper is a newly annotated corpus of multimodal Reddit posts. This dataset was collected from 15 subreddits related to emotion names and supplemented with data from the Pushshift Reddit Dataset. After initial collection and filtering, 1,054 posts were annotated by three human annotators for emotions, image-text relations, and emotion stimuli. This dataset is publicly available.

Key Insights and Ideas

  • Text is Often King (for Emotion): For many emotion categories, text alone is sufficient for accurate prediction. This is especially true for emotions like disgust, joy, and surprise.
  • Images Shine for Stimuli: The visual component is crucial for identifying the stimulus of an emotion. For many stimulus categories, unimodal image models or multimodal approaches outperform text-only models.
  • Complementarity is Key: The "complementary" image-text relation is the most dominant in the dataset, highlighting that images often provide essential context that text alone lacks for full emotional understanding.
  • Multimodal Models Excel in Nuance: While text is strong for basic emotion detection, multimodal models show significant improvements for certain emotions (like anger and sadness) and for complex stimulus categories (like art, events, memes, places, or screenshots).
  • Surprise and Disgust are Socially Linked: The co-occurrence analysis revealed interesting patterns, such as surprise frequently appearing with both positive and negative emotions, and a strong link between disgust and anger, possibly reflecting a "fight-or-flight" response to stimuli.

Key Results

  • Emotion Prediction: The text-based unimodal model achieved the highest F1 score (.53) for emotion prediction, outperforming the image-based model (.41). However, multimodal models (especially model-based fusion at .53) matched or slightly improved upon the text-only performance.
  • Stimulus Prediction: Multimodal models, particularly model-based fusion (.63), showed the best performance for stimulus detection, indicating the importance of combining both text and image information.
  • Relation Prediction: The text-based unimodal model also performed best for relation prediction (.77), with the "complementary" relation being the most frequent.
  • Specific Emotion Gains: Multimodal models were particularly beneficial for predicting anger and sadness, while text alone was sufficient for disgust, joy, and surprise.
  • Specific Stimulus Gains: Image-only models were best for stimuli like objects, animals, and food, while multimodal models excelled for art, events, memes, places, and screenshots.

Limitations and Future Work

  • Annotation Agreement: The inter-annotator agreement for relation classification was found to be low, suggesting subjectivity and potential challenges in this task.
  • Discrete vs. Dimensional Emotions: The study uses discrete emotion categories, which might not capture the full spectrum of emotional experience.
  • Future Directions: The authors suggest exploring segment-based open-domain stimulus detection combined with region-based image analysis to go beyond predefined categories. They also propose investigating how to ground textual references in specific image regions for a deeper understanding of multimodal emotion expression.

Summary updated

BibTeX
@inproceedings{khlyzova2022complementarity,
  author = {Anna Khlyzova and Carina Silberer and Roman Klinger},
  title = {{On the Complementarity of Images and Text for the Expression of Emotions in Social Media}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure