Understanding Annotator Perception: Modeling Psychological Inference from First- and Third-Person Annotations (Student Abstract)

Gerard Christopher Yeo, Kokil JaidkaAAAI 2025 · DOI

TL;DR

This paper investigates a critical assumption in large language model (LLM) training: that third-party annotations accurately reflect authors' true psychological states. By comparing first-person self-reports with third-person inferences from annotators, the study reveals significant discrepancies, particularly for nuanced cognitive appraisals. These findings challenge LLM training paradigms and highlight the need for models that better understand human intent.

What Problem Does This Paper Address?

Large language models (LLMs) are trained on massive datasets, often assuming that annotations within these datasets accurately represent the authors' genuine psychological states and intentions. However, this assumption overlooks the potential gap between an author's lived experience (first-person perspective) and how an external observer interprets that experience from text (third-person inference). This discrepancy is particularly problematic for applications that rely on LLMs to understand human emotions and intentions, such as sentiment analysis or personalized recommendations. The paper aims to quantify this gap and understand its implications for LLM performance.

Methods

The study employs a two-part data collection and analysis approach. First, they leverage the existing PEACE-Reviews dataset, which contains 1,400 self-annotated reviews of purchase experiences. These first-person annotations serve as the ground truth, detailing participants' emotions, cognitive appraisals, and behavioral intentions.

In the second part, they augment this dataset with third-person annotations. 365 participants from Amazon Mechanical Turk were recruited to rate the same review texts along various psychological and emotional dimensions. These annotators also provided ratings for "helpfulness" and "trustworthiness." The study then compares these third-person inferences against the first-person ground truth to assess the accuracy of external interpretations.

For modeling, they primarily use the Bidirectional Encoder Representations from Transformers (BERT) model. They conduct two main experimental setups:

  1. Baseline Modeling: Training classifiers using third-person annotations to predict various variables and comparing their performance against models trained on first-person annotations (from Yeo and Jaidka, 2023).
  2. Multi-task Modeling: Developing models that integrate third-person annotations to predict first-person labels, following a theoretical framework that considers cognitive appraisals and emotions to predict behavioral intentions.

Datasets

  • PEACE-Reviews dataset: A newly introduced dataset (Yeo and Jaidka, 2023) containing 1,400 self-annotated reviews of purchase experiences. This dataset provides first-person accounts of emotions, cognitive appraisals, and behavioral intentions, serving as the ground truth.
  • Third-person annotations: Collected by the authors for this study, these annotations supplement the PEACE-Reviews dataset. They consist of ratings from 365 Amazon Mechanical Turk workers on the same 1,400 reviews, covering psychological and emotional dimensions, as well as perceived helpfulness and trustworthiness.

Key Insights and Ideas

  • The "Intent Gap": There's a significant and measurable difference between how people describe their own psychological states and how others interpret those states from their writing. This isn't just a minor issue; it directly impacts the reliability of data used to train LLMs.
  • Nuance is Hard to Infer: While LLMs might be good at capturing broad emotional valence (positive/negative), they struggle with more complex cognitive appraisals, especially those involving external circumstances or future expectations. This suggests a limitation in how LLMs process subtle human reasoning.
  • First-Person Data is King (for now): Models trained on direct self-reports (first-person data) consistently outperform models trained on third-person inferences, particularly for variables like "goal conduciveness" and "intent to promote." This underscores the value of authentic, self-reported data for building accurate models of human experience.
  • Some Things are Easier to Guess: Interestingly, attributes like "Expectedness" were inferred better by third-party annotators than by the original writers themselves. This hints that some psychological states might be more implicitly expressed or universally understood, even if not explicitly articulated by the author.

Key Results

  • Models trained on first-person annotations (from Yeo and Jaidka, 2023) consistently outperform models trained on third-person annotations across various variables. For instance, in predicting "pleasantness," the first-person model achieved an F1 score of 0.74, compared to 0.56-0.58 for BERT and RoBERTa models using third-person data.
  • The drop in accuracy when using third-person data ranges from 5% to 11% for emotion prediction, and is also significant for other variables.
  • Multi-task models incorporating third-person cognitive appraisals and emotions showed improved performance in predicting intentions to promote and repurchase, with an F1 score of 0.64 for emotion prediction. However, these models still exhibited an accuracy drop of 6-11% compared to frameworks using first-person labels.
  • Appraisals related to valence (pleasantness, goal conduciveness) and fairness showed medium-to-large correspondence between first- and third-person ratings. Conversely, appraisals involving external agency (accountability) and future expectancy had lower correspondence.
  • Behavioral intentions were inferred better by third-party annotators than cognitive appraisals (correlation of r = .47).

Limitations and Future Work

Limitations: * The study highlights that third-party annotators struggle more with inferring cognitive appraisals, especially those involving external circumstances, compared to emotions and intentions. * While multi-task models show improvement, they still lag behind first-person-based models, indicating room for enhancement in third-person inference. * The paper notes that attributes like "Expectedness" are better inferred through third-person than first-person annotations, suggesting these might be difficult to express explicitly in text.

Future Work: * Investigate the underlying reasons for the observed discrepancies between first- and third-person perceptions. * Expand the analysis to other social media domains to assess the generalizability of findings. * Explore techniques to improve third-person inference accuracy, such as leveraging multi-modal approaches (e.g., incorporating visual or auditory cues) and context-aware annotation strategies.

Summary updated

BibTeX
@inproceedings{yeo2025understanding,
  author = {Gerard Christopher Yeo and Kokil Jaidka},
  title = {{Understanding Annotator Perception: Modeling Psychological Inference from First- and Third-Person Annotations (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35321},
}
Figure