Introduction
Presented at the 2023 Conference on Affective Computing and Intelligent Interaction (ACII), this research investigates a fundamental question in artificial intelligence: does the massive scale of Large Language Models (LLMs) grant them an "emergent" understanding of human emotion? By evaluating ChatGPT (GPT-3.5) on established affective computing benchmarks, the study demonstrates that these models can perform sophisticated sentiment analysis and adhere to complex psychological appraisal rules in a zero-shot capacity, often rivaling specialized, fine-tuned models.
The Challenge of Context in Affective Computing
Traditionally, the field of Affective Computing—the study of systems that recognize, interpret, and simulate human affect—has followed two primary paths:
- Rule-based Symbolic AI: Hard-coding psychological theories, such as the OCC model, into logical frameworks.
- Specialized Machine Learning: Training models on specific, labeled datasets for emotion detection.
A persistent hurdle for both approaches is the role of context. Understanding an emotion requires a nuanced grasp of the "who, what, and where" of a situation. This research explores whether LLMs, which are pre-trained on vast amounts of human discourse, can bridge this gap without explicit emotional training. Specifically, the authors test the model's grasp of the Valence-Arousal-Dominance (VAD) framework and the Ortony-Clore-Collins (OCC) appraisal model, which describes how cognitive evaluations of events lead to specific emotional states.
Methodology
The researchers employed instruction prompting (zero-shot) to evaluate ChatGPT-3.5 across three Research Questions (RQs):
- RQ1 (Sentiment Analysis): Can the model assign accurate VAD scores (0.0 to 1.0) to situational descriptions and emotion words?
- RQ2 (Affect Representation): Can the model map situations to specific emotion words using both numerical Euclidean distance (based on VAD scores) and latent semantic matching?
- RQ3 (Emotion Elicitation): Can the model act as an inference engine for the OCC model? The researchers "programmed" the logic-based OCC framework directly into a prompt to see if the model could predict emotions based on appraisal rules.
Datasets
The study utilized three primary benchmarks:
- ANET (Affective Norms for English Text): 120 situational descriptions with human-validated VAD ratings.
- Russell & Mehrabian Word List: 151 emotion words with associated ground-truth VAD scores.
- Custom Appraisal Scenarios: 12 scenarios designed to test specific branches of the OCC model (e.g., comparing reactions to success vs. failure).
Key Insights
Emergent Intelligence and Perspective
The model’s affective capabilities appear to be a byproduct of learning complex language patterns rather than a dedicated module. A critical finding involved the "Dominance" dimension (the feeling of being in control). While the model initially struggled, its accuracy improved significantly when the prompt clarified the perspective (e.g., "assess the control the main person feels"). This suggests that LLMs are highly sensitive to the subject of an emotional context.
Latent vs. Numerical Processing
ChatGPT demonstrated a superior ability to "feel" the connection between a situation and an emotion word through latent semantic representation rather than through explicit calculation. When forced to use its own numerical VAD scores to match words, performance was lower than when it was allowed to match them based on meaning alone.
Programmable Psychology
The success of the OCC experiment indicates that LLMs can serve as an "inference engine" for psychological theories. By "soft-coding" a theory of mind into a prompt, researchers can leverage the model's reasoning capabilities to simulate complex emotional appraisals.
Results and Performance
The model's performance on the 120 ANET situations showed remarkably strong Pearson correlations with human ground truth:
- Valence: $\rho = 0.95$
- Arousal: $\rho = 0.82$
- Dominance: $\rho = 0.82$
These results are comparable to, or exceed, state-of-the-art fine-tuned models like XLM-RoBERTa. Furthermore, using the prompted OCC rules, ChatGPT correctly predicted the emotion in 10 out of 12 complex scenarios, only struggling with the subtle distinction between "distress" and "disappointment."
Limitations and Future Directions
While the results are promising, the authors note several limitations. The study was conducted on GPT-3.5, and the rapid evolution of models like GPT-4 may yield different results. The model also exhibits a "semantic bias," occasionally prioritizing words that sound appropriate over those that are numerically accurate in VAD space.
Finally, the study focused on "snapshots" of emotion. Future research is required to determine if LLMs can simulate emotional dynamics over time—such as mood decay or transitions between emotional states—and whether these capabilities remain consistent across diverse, multi-modal, and cross-cultural datasets.
Summary updated