Personal Bias in Prediction of Emotions Elicited by Textual Opinions

Piotr Milkowski, Marcin Gruza, Kamil Kanclerz, Przemyslaw Kazienko, Damian Grimling, Jan KoconACL 2021

TL;DR

This paper tackles the challenge of personalized emotion recognition from text, moving beyond average user sentiment. They introduce "Personal Emotional Bias" (PEB), a novel measure that captures an individual's unique emotional perception based on their past annotations. This PEB, combined with a BERT-based architecture, significantly improves the accuracy of predicting emotions for individual users, paving the way for more nuanced and personalized NLP applications.

What Problem Does This Paper Address?

Traditional sentiment analysis and emotion recognition models often rely on aggregated annotations from multiple users. While this approach is effective for understanding general population sentiment, it falls short when trying to predict the emotions evoked by text in a specific individual. People's emotional responses are subjective and influenced by a myriad of factors, including personal experiences, beliefs, and even demographic background. This inherent subjectivity leads to significant differences between individual emotional responses and the "average" response captured by aggregated datasets. The paper aims to bridge this gap by developing a method that accounts for this individual variability, enabling more personalized emotion prediction.

Methods

The core of the paper's contribution lies in two key areas:

  • Personal Emotional Bias (PEB) Measure: The authors propose a novel metric, PEB(u, c), to quantify an individual user's (u) unique emotional perception for a specific category (c). This measure is calculated by comparing a user's past annotations to the average annotations of all other users for the same texts. Essentially, it captures how much an individual's emotional response deviates from the norm.
  • BERT-based Transformer Architecture: They develop a new BERT-based architecture that incorporates the PEB measure as an additional feature alongside text embeddings. This allows the model to leverage both the semantic content of the text and the user's historical emotional tendencies for more accurate predictions.

The study also explores two reasoning approaches: multi-task classification (predicting discrete emotion categories) and multivariate regression (predicting numerical emotion values).

Datasets

The primary dataset used in this research is the Sentimenti dataset, specifically the CAWI2 subset. This dataset consists of approximately 7,000 internet reviews, each annotated by around 50 individuals. The annotations cover ten emotional categories: valence, arousal, and eight basic emotions from Plutchik's model (sadness, anticipation, joy, fear, surprise, disgust, trust, and anger). The paper also mentions using translated versions of Polish texts into eight other languages for multilingual experiments.

Key Insights and Ideas

  • Individual responses vary significantly: The study strongly confirms that individual emotional responses to the same text can differ substantially from the average, highlighting the limitations of purely aggregated approaches.
  • PEB is a powerful predictor: The Personal Emotional Bias (PEB) measure proves to be a highly effective feature for improving personalized emotion recognition, often outperforming models that only use text content or demographic data.
  • Few annotations are needed for PEB: Surprisingly, the research indicates that even a small number of past annotations (around 5-7 texts) from a user can be sufficient to capture their personal biases and significantly improve prediction quality.
  • PEB and text are complementary: The PEB measure and text-based features are shown to be complementary. When one performs poorly, the other often performs well, and their combination yields the best results. This suggests that personal history and textual content offer distinct but valuable information for emotion prediction.
  • Controversy matters: Computing PEB using the most controversial texts (those with the highest disagreement among annotators) can yield greater benefits. This implies that understanding how an individual reacts to ambiguous or divisive content is particularly informative.

Key Results

The paper demonstrates significant improvements in emotion recognition accuracy when incorporating PEB.

  • Classification: In classification tasks, the TXT+PEB scenario consistently outperformed the baseline TXT scenario, achieving substantial gains in F1-macro scores. For instance, with the HerBERT model, the TXT+PEB scenario reached an average F1-macro of 37.85%, compared to 17.69% for the TXT scenario. The ALL scenario (including demographic data) showed only marginal improvements over TXT+PEB, suggesting PEB is the dominant personalization factor.
  • Regression: Similarly, in regression tasks, the TXT+PEB scenario significantly improved R-squared values compared to the TXT baseline. The average R-squared for TXT+PEB was 44.93%, while the TXT baseline was only 14.08%.
  • Multilingual Performance: The multilingual experiments using XLM-RoBERTa showed that the TXT+PEB approach also yielded strong results across various languages, indicating the generalizability of the PEB concept.

Limitations and Future Work

  • Data Imbalance: The authors acknowledge data imbalance in certain emotional categories, which can affect model performance. They used F1-macro to mitigate this in classification.
  • Future Work: The authors propose several avenues for future research:
  • Investigating the impact of individual components of the PEB vector on recognition quality.
  • Extending PEB with information about the average annotation value of texts.
  • Comparing the quality of dedicated models for individual emotional dimensions against the multi-task model.
  • Exploring the impact of PEB on different types of texts and emotional dimensions.

Summary updated

BibTeX
@inproceedings{milkowski2021personal,
  author = {Piotr Milkowski and Marcin Gruza and Kamil Kanclerz and Przemyslaw Kazienko and Damian Grimling and Jan Kocon},
  title = {{Personal Bias in Prediction of Emotions Elicited by Textual Opinions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure