The CARE Dataset for Affective Response Detection

Jane A. Yu, Alon Y. Halevy

Abstract

Social media plays an increasing role in our communication with friends and family, and our consumption of information and entertainment. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective response to a post (e.g., whether the user is likely to be humored, inspired, angered, informed). Similar to work on emotion recognition (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data. We introduce CARE$_{db}$, a dataset of 230k social media posts annotated according to 7 affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response of the readers to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowd-sourced annotations. Finally, we use CARE$_{db}$ to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.

Introduction

Understanding how content makes a user feel—the affective response—is crucial for effective social media ranking and recommendation. Traditional emotion recognition focuses on the publisher's affect, which often differs from the reader's response. For instance, an angry political post might elicit "sadness" or "approval" depending on the reader. Historically, capturing these responses required expensive manual annotation. To address this, researchers have introduced CARE (Common Affective Response Expression), an unsupervised method that leverages user comments to automatically label posts with the emotional responses they elicit.

The CARE Method

The CARE method operates on the principle that user comments often explicitly state a reader's response (e.g., "This is so funny!"). The process involves four key stages:

  1. Pattern Matching: Using seed regular expressions and a lexicon of indicators (e.g., "funny" maps to amused).
  2. Iterative Expansion: A bootstrapping approach identifies common n-grams in labeled comments to grow the lexicon and pattern set.
  3. Aggregation: To ensure high precision, a post is only labeled if at least five unique comments express the same affect.
  4. CARE-BERT Training: A BERT-base model is trained on post text using these automated labels, allowing it to predict responses for new posts without comments.
The CARE method pipeline: extracting indicators from comments, labeling affect, aggregating responses to posts, and iteratively expanding the lexicon.

Datasets and Validation

The primary output is CAREdb, a dataset of 230,000 Reddit posts annotated across seven categories: adoring, amused, approving, excited, angered, saddened, and scared.

Validation against human annotators showed a 94% agreement rate for at least one label, confirming that the unsupervised aggregation of comments acts as a high-precision "wisdom of the crowd." Furthermore, analysis reveals that publisher affect and reader response only intersect about 44% of the time, underscoring the need for reader-centric labeling rather than relying on author-centric emotion detection.

Key Results

CARE-BERT achieves a micro-F1 score of 0.69 on predicting responses from post text. As shown in the performance analysis, the model significantly outperforms state-of-the-art emotion classifiers like GoEmotions when applied to affective response tasks, particularly as the training set size increases.

F1-score comparison across different training set sizes, showing CARE-BERT (no_freeze) outperforming baselines and the GoEmotions model.

Beyond its primary task, CARE-BERT demonstrates strong transfer learning capabilities. When fine-tuned on the ISEAR dataset, it performs competitively with models trained on much larger manually labeled datasets, suggesting that CAREdb captures deep semantic nuances of human emotion that are useful for broader NLP tasks.

Conclusion and Future Work

The CARE method provides a scalable, high-precision way to label social media content without human intervention. While currently limited by regex-based extraction (which can struggle with sarcasm or emojis) and a dependency on comment volume, the researchers plan to extend the method to multi-modal content like images and memes, where visual context is critical to emotional impact.

Summary updated

Abstract

Social media plays an increasing role in our communication with friends and family, and our consumption of information and entertainment. Hence, to design effective ranking functions for posts on social media, it would be useful to predict the affective response to a post (e.g., whether the user is likely to be humored, inspired, angered, informed). Similar to work on emotion recognition (which focuses on the affect of the publisher of the post), the traditional approach to recognizing affective response would involve an expensive investment in human annotation of training data. We introduce CARE$_{db}$, a dataset of 230k social media posts annotated according to 7 affective responses using the Common Affective Response Expression (CARE) method. The CARE method is a means of leveraging the signal that is present in comments that are posted in response to a post, providing high-precision evidence about the affective response of the readers to the post without human annotation. Unlike human annotation, the annotation process we describe here can be iterated upon to expand the coverage of the method, particularly for new affective responses. We present experiments that demonstrate that the CARE annotations compare favorably with crowd-sourced annotations. Finally, we use CARE$_{db}$ to train competitive BERT-based models for predicting affective response as well as emotion detection, demonstrating the utility of the dataset for related tasks.
BibTeX
@inproceedings{yu2022care,
  author = {Jane A. Yu and Alon Y. Halevy},
  title = {{The CARE Dataset for Affective Response Detection}},
  booktitle = {CoNLL},
  year = {2022},
  doi = {10.18653/v1/2022.conll-1.5},
}
Figure