Transformer-based Prediction of Emotional Reactions to Online Social Network Posts

Irene Benedetto, Moreno La Quatra, Luca Cagliero, Luca Vassio, Martino TrevisanACL 2023

TL;DR

This paper introduces a novel Transformer-based approach to predict the number of emotional reactions a social media post will receive before it's published. By leveraging advanced language understanding and incorporating post metadata, their method significantly outperforms traditional techniques and provides interpretable explanations for its predictions.

What Problem Does This Paper Address?

Understanding how users react to online content is crucial for marketers, researchers, and platform developers. While predicting the popularity of a post (e.g., number of likes, views) has been studied, predicting the specific emotional reactions (like "love," "angry," "wow") is more nuanced and challenging. Existing methods often rely on basic text statistics (like TF-IDF) that miss semantic nuances and use "black box" models that offer no insight into why a prediction was made. This lack of explainability limits trust and actionable insights. The authors aim to bridge this gap by developing a model that not only predicts reaction counts but also explains its reasoning.

Methods

The core of the proposed solution is a Transformer-based architecture named Transformer+Metadata. Here's a breakdown:

  • Transformer Encoder: They utilize XLM-RoBERTa, a powerful pre-trained Transformer model, to deeply understand the textual content of a post. This allows it to capture complex semantic relationships that simpler methods miss.
  • Metadata Integration: Crucially, the model also incorporates post metadata (like author's follower count, publication time, presence of links/images). This is done through an "average pooling" operation on the text representation and a "fusion stage" where text and metadata are combined.
  • Multi-task Regression: The model is trained as a multi-task regressor, predicting the count for each of the 7 distinct emotional reaction types simultaneously. To handle skewed distributions, they predict the logarithm of reaction counts.
  • Explainability with SHAP: To address the "black box" problem, they employ the SHAP (SHapley Additive exPlanations) framework. SHAP values are used to quantify the contribution of individual words (tokens) in the post's text to the predicted reaction count, providing interpretable insights.

They compare their Transformer+Metadata model against several baselines: Metadata-Only (using traditional ML models on metadata), TF-IDF+Metadata (combining TF-IDF text features with metadata), and TF-IDF (text features only).

Datasets

The study uses a large dataset of posts collected from the Facebook social network.

  • Source: Data was gathered using the CrowdTangle platform and its API.
  • Content: Posts from profiles of UK newspapers and TV/radio stations of national and local importance were included.
  • Size: Approximately 6 million posts were collected over three years (early 2019 to mid-2022).
  • Filtering: Influencers with fewer than 100k followers were excluded.
  • Processed Data: A total of 709,142 posts were processed and split into training, testing, and development sets (80/10/10 ratio).
  • Features: Each post included a textual caption (max 500 characters), and optionally, the title and first sentence of linked webpages. Metadata included creation time and historical reaction data.

Key Insights and Ideas

  • Transformers Excel at Semantics: The most significant insight is that Transformer-based models, particularly XLM-RoBERTa, are far superior at capturing the semantic nuances of text for predicting emotional reactions compared to traditional TF-IDF methods.
  • Metadata's Limited Impact (for Transformers): While metadata is important for simpler models, its impact on the Transformer-based models is surprisingly limited. The improvements of Transformer+Metadata over Transformer-Only (text only) are often less than 1 percentage point, suggesting the text encoding is the dominant factor.
  • Explainability is Key: The SHAP analysis reveals that specific words and phrases are indeed strongly correlated with particular emotional reactions. This provides actionable insights for content creators.
  • Reaction-Specific Language: The paper highlights how different reactions are associated with distinct linguistic patterns. For instance, "care" reactions are linked to words about patience and support, while "angry" reactions correlate with terms related to victimhood or injustice.

Key Results

  • Performance Improvement: Transformer-based models (Transformer-Only and Transformer+Metadata) significantly outperform all baseline models across most reaction types, as measured by Median Absolute Percentage Error (MedAPE).
  • Best Model: Transformer+Metadata achieved the best performance for most reactions, with a MedAPE of 45.26% for "Wow" reactions.
  • Transformer-Only Competitiveness: Transformer-Only (text only) performed nearly as well as Transformer+Metadata, often with statistically insignificant differences, reinforcing the power of text alone.
  • Rare Reactions: For reactions like "Care," which often have zero occurrences, Transformer-based models achieve very low MedAPE values (e.g., 0.05% for Transformer+Metadata), indicating they can accurately predict the absence of a reaction.
  • Explainability Examples: SHAP analysis identified specific tokens like "victim," "surgery," and "stress" as highly influential for "Angry" and "Sad" reactions, and "successful," "care," and "scientist" for "Love" reactions.

Limitations and Future Work

  • Generalizability: The study was conducted on Facebook posts from news and media outlets. The authors acknowledge that the methodology might not generalize perfectly to other social networks or domains. Further evaluation on diverse datasets is needed.
  • Feature Scope: The model relies solely on textual content and metadata. Future work could explore incorporating other factors like user demographics or multimedia content (images, videos) to potentially improve prediction accuracy.
  • Depth of Explanation: While SHAP provides valuable insights into token influence, it may not fully capture the complex psychological reasons behind user emotional responses.
  • Dataset Bias: The dataset consists of public posts from specific types of profiles. The authors note that the dataset was collected in a lawful and ethical environment, using only public data.

Summary updated

BibTeX
@inproceedings{benedetto2023transformer,
  author = {Irene Benedetto and Moreno La Quatra and Luca Cagliero and Luca Vassio and Martino Trevisan},
  title = {{Transformer-based Prediction of Emotional Reactions to Online Social Network Posts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure