In affective computing, the "gold standard" for training models has traditionally been the consensus of external observers. If five people look at a photo or read a tweet and agree it looks "angry," the model is taught that the ground truth is anger. However, a growing body of research suggests this approach ignores a fundamental psychological reality: what we express to the world is often a muted, filtered, or even deceptive version of what we actually feel.
Researchers are now shifting toward author-centric modeling, which prioritizes the internal state of the subject (the "intended" emotion) over the external interpretation of the audience (the "perceived" emotion). This shift reveals a systematic "perception gap" that current AI models struggle to bridge.
The Systematic Underestimation of Intensity
One of the most consistent findings across this research area is that external observers are conservative judges of emotion. In a foundational study, Carlos Busso and Shrikanth S. Narayanan found that speakers perceive their own emotions as significantly more specific and intense than listeners do The expression and perception of emotions: comparing assessments of self versus others.
This isn't just a minor disagreement; it’s a systematic bias. Data from the WRIME dataset, a collection of 17,000 Japanese social media posts, shows that readers (and the AI models trained on them) consistently underestimate the intensity of a writer’s anger and trust WRIME: A New Dataset for Emotional Intensity Estimation with Subjective and Objective Annotations. When an author feels a "10" on an emotional scale, an observer might only see a "6." Models trained exclusively on observer labels essentially learn to "turn down the volume" on human experience, failing to capture the true magnitude of the subject's internal state.
The Perception Gap in Sarcasm and Intent
The discrepancy between intent and perception is perhaps most visible in complex linguistic phenomena like sarcasm. Traditional datasets often use third-party annotators to guess if a tweet is sarcastic. However, Silviu Oprea and Walid Magdy demonstrated that models trained on these "perceived" labels fail significantly when tested against "intended" sarcasm—labels provided by the original authors themselves iSarcasm: A Dataset of Intended Sarcasm.
The reason is context. An author knows their own history, personality, and relationship with the audience. Without this "author context," observers rely on superficial markers or stereotypes to guess intent Exploring Author Context for Detecting Intended vs Perceived Sarcasm. As Jiayi Li and colleagues argue, relying on third-party labels may inadvertently train models to replicate social biases rather than identify genuine emotional reality Can Third Parties Read Our Emotions?.
Invisible States and Private Emotions
Some internal states are almost entirely invisible to the outside world. Cognitive load (mental effort) and "private" emotions like relief or admiration are notoriously difficult for observers to detect through facial expressions or voice alone.
- Cognitive Load: In the AVCAffe dataset, researchers found that while external observers struggle to judge mental demand, participant self-assessments provide a much clearer signal for building models that can quantify "Zoom fatigue" AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work.
- Subtle Affect: In customer service simulations, Thomas Demeester and colleagues noted that feelings like relief are often "invisible" to third parties, leading to a total disconnect between how a customer feels and how a support agent (or an AI) perceives the interaction A Customer Journey in the Land of Oz.
Toward Author-Centric Ground Truth
To solve these issues, the field is moving toward large-scale datasets where the "ground truth" is provided by the person experiencing the emotion.
- Organic Self-Labeling: The Vent dataset leverages social media platforms where users explicitly tag their posts with their feelings. Anton V. Malko and colleagues have validated that these self-assigned labels are highly reliable indicators of emotional state, providing a massive resource for training more authentic models Demonstrating the Reliability of Self-Annotated Emotion Data.
- Real-Time Probing: Recent work by Cole Johnson and Shrikanth Narayanan introduces "author labeling," where document creators annotate their own text in real-time via LLM-generated questions. This method captures intent that observers miss more than half the time Authors Should Label Their Own Documents.
- Dual-Perspective Modeling: Datasets like WRIME and iSarcasm are unique because they provide both perspectives. This allows researchers to train models that don't just predict one or the other, but actually model the relationship between expression and perception.
The Trade-off: Subjectivity vs. Consistency
While self-reports capture the "true" internal state, they are not a silver bullet. Helen Schneider and colleagues point out a fundamental trade-off: external observers offer higher consistency and lower individual bias, whereas self-assessments are prone to memory errors and social signaling Datasets for Valence and Arousal Inference: A Survey. For instance, users on social media may curate their "expressed" emotions to fit a public persona, which might still differ from their "felt" psychological state Multimodal Sentiment Analysis To Explore the Structure of Emotions.
Ultimately, the goal of this research direction is to move AI beyond being a "polite observer" that understands social cues, toward being an "empathetic partner" that understands the human behind the words. By modeling the gap between what we say and what we feel, AI can begin to account for the nuance of human intent.