For decades, affective computing and natural language processing have operated under a convenient assumption: that human emotions and internal states can be objectively measured by outside observers. In practice, this means datasets are typically labeled by third-party crowdworkers who read a tweet, watch a video, or listen to an audio clip and guess what the subject is feeling.
However, a growing body of research reveals a fundamental flaw in this paradigm: external consensus is a poor proxy for internal reality. As Carlos Busso et al. show in The expression and perception of emotions: comparing assessments of self versus others, there is a significant mismatch between a speaker's internal emotional intent and how external observers perceive that emotion. Speakers consistently experience their own emotions with greater specificity and intensity than outside listeners can detect.
When we train machine learning models on third-party annotations, we are not teaching them to recognize actual human emotions. Instead, as Jiayi Li et al. argue in Can Third Parties Read Our Emotions?, we are training them to replicate external perceptions, social stereotypes, and observer biases. To build systems that truly understand human psychology, the field is shifting toward self-reported ground truth—capturing the "private states" of the individuals experiencing the emotions.
The Systematic Distortions of Third-Party Labels
The distortion introduced by third-party labels is not just random noise; it follows systematic patterns that fundamentally limit model performance.
The Dampening Effect
External observers tend to be highly conservative when estimating someone else's feelings. In a study on conversational data, Sofie Labat et al. found in Variation in the Expression and Annotation of Emotions: A Wizard of Oz Pilot Study that third-party annotators systematically assign more neutral or moderate labels compared to the intense self-reports of the individuals experiencing the interaction. This systematic dampening means models trained on observer data consistently underestimate the emotional depth of user experiences.
The Missing "Why" (Cognitive Appraisals)
Emotions do not occur in isolation; they are driven by internal cognitive evaluations—such as fairness, goal consistency, or personal motivation. Third-party annotators cannot accurately reconstruct these internal states. For instance, Xu Chen et al. note in REASONER: An Explainable Recommendation Dataset with Comprehensive Labeling Ground Truths that relying on external annotators to infer the reasons behind a user's preference introduces substantial noise. Without authentic ground truth, models cannot learn individual motivations.
This gap becomes critical when trying to predict future behavior. Standard NLP models can easily classify a product review as "happy" or "sad," but they struggle to predict whether that customer will actually buy from the company again. As Gerard Christopher Yeo et al. demonstrate in Beyond Text: Leveraging Multi-Task Learning and Cognitive Appraisal Theory for Post-Purchase Intention Analysis and The PEACE-Reviews dataset: Modeling Cognitive Appraisals in Emotion Text Analysis, models must capture the author's true internal cognitive appraisals to bridge the gap between expressed text and future action.
The Empathy Disconnect
Complex, subjective social phenomena like empathy are particularly vulnerable to third-party distortion. Datasets tracking empathy frequently rely on external observers, which creates a severe disconnect from the actual internal experience of the writer (Allison Lahnala et al. in CAISA at WASSA 2022: Adapter-Tuning for Empathy Prediction). As Jocelyn Shen et al. emphasize in EmpathicStories++: A Multimodal Dataset for Empathy Towards Personal Experiences, empathy is an inherently subjective internal state; third-party labeling forces a mismatch between training data and the user's actual psychological reality.
Designing for First-Person Ground Truth
To overcome these limitations, researchers are building datasets where the ground truth comes directly from the source.
In Modeling Empathy and Distress in Reaction to News Stories, Sven Buechel et al. established a new gold standard by collecting self-reported emotional ratings directly from authors. This approach successfully captured subtle linguistic markers of internal experience that external observers routinely missed. Similarly, Tomoyuki Kajiwara et al. introduced the WRIME dataset in WRIME: A New Dataset for Emotional Intensity Estimation with Subjective and Objective Annotations, explicitly pairing subjective (writer) and objective (reader) annotations. Their analysis highlighted a massive "empathy gap," showing that outside readers and AI models consistently struggle to detect the true intensity of a writer's internal trust or anger.
By centering the original narrator's perspective, models can begin to capture deep emotional resonance. Jocelyn Shen et al. leveraged this in Modeling Empathic Similarity in Personal Narratives (and its parallel work Modeling Empathic Similarity in Personal Narratives), demonstrating that models trained on first-person validation can identify stories that emotionally resonate with users across entirely different topics, far outperforming standard semantic search. Furthermore, Mia Huong Nguyen et al. show in EMO-KNOW: A Large Scale Dataset on Emotion-Cause that utilizing self-reported emotional states is essential for capturing the authentic "why" behind emotions, reducing the inherent noise of third-party perception.
Alternative Paradigms: Behavior and Intentionality
While self-reporting is a powerful tool, it introduces its own challenges. For example, during retrospective reviews, individuals sometimes struggle to accurately recall their past emotional states, occasionally interpreting their own facial expressions as an outside observer would (Shazia Afzal et al. in Intentional Affect: An Alternative Notion of Affective Interaction with a Machine). To move beyond inferential annotation entirely (Tatsuya Ide et al. in Building a Dialogue Corpus Annotated with Expressed and Experienced Emotions), researchers are exploring two alternative paths:
- Objective Behavioral Experiments: Instead of asking humans to subjectively annotate text or internal states, Vasudha Varadarajan et al. propose in Capturing Human Cognitive Styles with Language: Towards an Experimental Evaluation Paradigm linking language patterns directly to objective behavioral experiments (like classic decision-making tasks). This provides a rigorous psychological ground truth for cognitive styles without relying on perception-biased labels.
- Intentional Affect: Rather than trying to covertly "detect" hidden internal emotions, Shazia Afzal et al. suggest designing systems for Intentional Affect—a paradigm where users deliberately use expressions to communicate with a machine they know is listening, transforming emotion from a passive hidden state into an active communicative intent.
Why It Works: Changing the Optimization Target
To understand why this shift matters, consider what a machine learning model actually optimizes for during training.
When a model is trained on third-party annotations, its loss function minimizes the distance between its predictions and average human perception. The model learns to look for obvious, stereotypical surface features—like exclamation points or explicit sentiment words—because that is what external crowdworkers rely on to guess an emotion. It learns to replicate human social stereotypes rather than internal reality.
[Text/Expression] ---> [Third-Party Annotator] ---> [Stereotyped Label]
^
| (Model optimizes to replicate this perception)
When a model is trained on self-reported ground truth or objective behavioral data, the optimization target changes completely. The model is forced to find the subtle, often counter-intuitive linguistic and contextual markers that map directly to an individual's actual internal state or subsequent actions.
[Text/Expression] ---> [Internal Private State] ---> [Self-Reported Label]
^
| (Model optimizes to discover true internal reality)
For example, a writer might use highly restrained language while experiencing intense internal anger. A third-party annotator would label this text as neutral, but the writer labels it as highly intense. By learning from the writer's own label, the model discovers the hidden, non-obvious features of true emotional intensity. By replacing the distorted lens of the external observer with direct, first-person ground truth, models move past surface-level perception and begin to capture the authentic psychological reality of human interaction.
Key Foundational Work
The expression and perception of emotions: comparing assessments of self versus others (2008) — contributed 2 concept elements to this direction.
Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data (2008) — contributed 1 concept elements to this direction.
Intentional Affect: An Alternative Notion of Affective Interaction with a Machine (2009) — contributed 1 concept elements to this direction.