TL;DR
This paper introduces NEmo+, a large-scale dataset for predicting emotional responses to multi-modal news (headline + image), focusing on the sensitive topic of gun violence. They develop and benchmark models for both predicting a single dominant emotion and the distribution of emotions evoked by news content, showing significant improvements over baselines.
What Problem Does This Paper Address?
Understanding how news presentation influences people's emotional reactions is crucial for combating manipulation and fostering media literacy. While sentiment analysis is common, predicting specific emotional responses to news content, especially when combining text and images, is a less explored area. Existing work often focuses on single modalities or doesn't capture the nuanced emotional spectrum. This paper tackles the challenge of predicting emotional responses to multi-modal news, specifically in the context of gun violence, a highly polarizing and emotionally charged topic.
Methods
The authors introduce the NEmo+ dataset, an extension of BU-NEmo, containing 1,297 news headline and lead image pairs with extensive annotations. They develop two main approaches:
- Single-label prediction on consensus data: For news items that evoke a clear consensus in emotional response, they train models to predict a single dominant emotion. This involves filtering the dataset to create NEmo+-CR (Consensus by Rank) and NEmo+-CE (Consensus by Entropy). They experiment with BERT (A-BERT) for text and ResNet (A-ResNet) for images, including transfer learning from the ArtEmis dataset. For image-only prediction, they also explore converting images to text using web entity tags or automatic captioning.
- Multi-label prediction on the full dataset: To address news items that elicit diverse emotional responses, they frame the problem as multi-label classification. They convert the frequency distribution of emotions into binary labels based on a threshold and use BERT for this task across text, image-converted-to-text, and multi-modal inputs.
Datasets
- NEmo+: A newly introduced dataset, significantly expanding upon BU-NEmo. It contains 1,297 news headline and lead image pairings related to gun violence, with 38,910 annotations.
- BU-NEmo: The predecessor dataset, containing 320 news items.
- ArtEmis: Used for transfer learning. This dataset contains art-related images with emotional annotations.
- NEmo+-Consensus (NEmo+-CR and NEmo+-CE): Filtered subsets of NEmo+ containing news items with high emotional consensus, used for single-label prediction.
Key Insights and Ideas
- Multi-modal news significantly impacts emotional response: The study highlights that combining headlines and images can evoke different emotional reactions than either modality alone.
- Images can be ambiguous without context: For news images, especially those not inherently emotional like artwork, textual context (from headlines or tags) is crucial for eliciting predictable emotional responses.
- Emotional consensus is not always present: A significant portion of news items (over 60%) in NEmo+ elicit diverse emotional reactions, necessitating multi-label prediction approaches.
- Transfer learning from related domains is beneficial: Pre-training models on the ArtEmis dataset improved performance, particularly for text-based models.
- Gun violence news tends to evoke negative emotions: The dataset distribution shows a clear inclination towards negative emotions like sadness and fear, which is important for baseline comparisons.
Key Results
- Single-label prediction: The best model (A-BERT on NEmo+-CR) achieved significant improvements over random and majority baselines, outperforming them by over 55 and 26 percentage points respectively for the image-only condition. Image-based models, when converted to text, outperformed text-only models.
- Multi-label prediction: The models consistently outperformed simulated random baselines across all conditions and thresholds, showing substantial gains in exact match accuracy and LRAP scores, especially at higher thresholds.
- TI (Text + Image) performance: Surprisingly, the multi-modal TI models showed lower performance than single-modal models in the consensus dataset, which the authors attribute to the limited dataset size and potential overfitting.
Limitations and Future Work
- Limited multi-modal performance: The TI multi-modal model's lower performance is attributed to the small size of the NEmo+-Consensus dataset, making it difficult to learn robust multi-modal features.
- Loss of information in multi-label conversion: Converting emotion distributions to binary labels for multi-label prediction discards information about the relative scale of emotions.
- Future directions:
- Model the distribution of emotions using KL-Divergence loss instead of binary labels.
- Utilize intensity scores to predict the strength of emotional responses.
- Predict whether a news item will elicit emotional consensus or division.
- Study the relationship between emotional responses and news framing.
- Extend the task to multilingual settings.
- Make the benchmarked systems accessible to researchers.
Summary updated