That's so cute!: The CARE Dataset for Affective Response Detection

Jane A. Yu, Alon Y. Halevy

TL;DR

This paper introduces CAREdb, a novel dataset of 230,000 social media posts annotated for affective response, derived using an unsupervised method that leverages comment signals. They also present CARE-BERT, a model trained on this dataset that achieves competitive performance in predicting affective responses, demonstrating the utility of their approach for both dataset creation and downstream tasks.

What Problem Does This Paper Address?

Social media platforms are increasingly central to how we communicate and consume information. To improve content ranking and user experience, it's crucial to understand how users feel about a post – their affective response (e.g., amused, inspired, angered). Traditional methods for this require expensive human annotation. Existing emotion detection focuses on the publisher's emotion, which is only a partial signal for the viewer's affective response. This paper tackles the challenge of creating a large-scale dataset for affective response detection without relying on costly human labeling, and then using it to build effective predictive models.

Methods

The core innovation is the Common Affective Response Expression (CARE) method. It works by:

  1. Extracting Affect from Comments: CARE uses predefined patterns and a keyword-affect lexicon to identify expressions within comments that signal a reader's affective response to a post. For example, "This is so funny" indicates an "amused" response.
  2. Aggregating and Labeling Posts: The affective responses from multiple comments on a post are aggregated. A post is assigned an affective response label if a sufficient number of comments (a threshold t, set to 5 in experiments) express that affect.
  3. Iterative Expansion: The system is bootstrapped with an initial set of patterns and a lexicon. It then iteratively expands its capabilities by identifying frequent n-grams in comments associated with specific affective responses, which are then used to enrich the lexicon or generate new CARE patterns.

This unsupervised approach allows for scalable annotation and the addition of new affective response categories.

Datasets

  • CAREdb: The primary contribution is this newly created dataset, containing 230,000 social media posts (from Reddit) annotated with 7 affective responses. It was built using the CARE method.
  • Human-annotated data: A subset of 6,000 posts from CAREdb was also annotated by human crowd-workers (Amazon Mechanical Turk Masters) for validation.
  • GoEmotions dataset: Used for comparison and transfer learning experiments, this dataset contains fine-grained emotion labels for Reddit comments.
  • ISEAR Dataset: Used for transfer learning experiments to evaluate the model's ability to predict publisher affect with limited labeled data.

Key Insights and Ideas

  • Comments as a Rich Signal: The paper convincingly demonstrates that comments on social media posts provide a high-precision, albeit indirect, signal for the affective response to the post itself. This bypasses the need for direct user feedback or publisher emotion analysis.
  • Unsupervised Annotation is Feasible and Scalable: The CARE method offers a practical way to build large-scale affective response datasets without the prohibitive cost of human annotation, making it adaptable to new categories and domains.
  • Affective Response vs. Publisher Emotion: The work highlights that affective response (how a viewer feels) is distinct from publisher emotion (how the poster feels), with only a moderate correlation (44% average intersection). This underscores the need for dedicated affective response detection.
  • CARE-BERT's Versatility: The trained CARE-BERT model not only predicts affective response effectively but also shows promise for transfer learning to related tasks like emotion detection, even with limited labeled data.

Key Results

  • CAREdb Creation: A dataset of 230,000 posts with 7 affective response labels was created.
  • Human Evaluation Agreement: CARE annotations showed high agreement with human annotators (90% of posts had all CARE labels confirmed by at least two out of three annotators).
  • CARE-BERT Performance: CARE-BERT achieved competitive performance for affective response prediction, comparable to SOTA models for emotion detection.
  • Transfer Learning Success: CARE-BERT, when fine-tuned on the ISEAR dataset for emotion detection, performed comparably to models trained on the GoEmotions dataset, demonstrating its utility for related tasks with limited data.
  • Publisher vs. Viewer Affect: The study found only a 44% average intersection between affective response and publisher affect labels, emphasizing their distinct nature.

Limitations and Future Work

  • Low Recall for Certain Classes: The CARE method, while precise, can have lower recall for certain affective responses, particularly "amused" and "excited." The authors suggest tuning thresholds per class or exploring alternative mechanisms.
  • Domain and Cultural Sensitivity: The authors acknowledge the need to test for cultural biases and re-evaluate the method when applied to new domains to ensure reliable performance and prevent unintended consequences like user manipulation.
  • Handling Ambiguity: Future work could address cases where indicators map to multiple affects or where common phrases for an affect are absent.
  • Incorporating More Signals: The authors suggest extending CARE to include emojis, negations, punctuation, and to investigate its application to images and multi-modal content like memes.
  • Ensembling: Exploring ensembling approaches with other emotion detection models is suggested to potentially improve performance and cover more classes.

Summary updated

Abstract

Social media plays an increasing role in our communication with friends and family, and in our consumption of entertainment and information. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective responses of a post (e.g., whether it is likely to elicit feelings of entertainment, inspiration, or anger). Similar to work on emotion detection (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data. We create and publicly release CARE DB, a dataset of 230k social media post annotations according to seven affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowdsourced annotations. Finally, we use CARE DB to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.
BibTeX
@inproceedings{yu2022that,
  author = {Jane A. Yu and Alon Y. Halevy},
  title = {{That's so cute!: The CARE Dataset for Affective Response Detection}},
  booktitle = {CoNLL},
  year = {2022},
  doi = {10.18653/v1/2022.conll-1.5},
}
Figure