Recent Advancement of Emotion Cognition in Large Language Models

Yuyan Chen, Yanghua XiaoarXiv 2024 · DOI · arXiv

Abstract

This paper provides a comprehensive survey of the emerging field of "emotion cognition" in Large Language Models (LLMs). By mapping recent research to the seven stages of human cognition defined by psychologist Ulric Neisser—Sensation, Perception, Imagination, Retention, Recall, Problem-Solving, and Thinking—the authors establish a unified framework for understanding how LLMs process human emotions. The survey highlights the evolution of LLMs from simple sentiment classifiers into sophisticated agents capable of empathy and Theory of Mind, while identifying critical bottlenecks in originality, cross-cultural generalization, and the "Clever Hans" effect in social reasoning.

Introduction: The Challenge of Emotional Intelligence

While Large Language Models (LLMs) have achieved remarkable success in general linguistic tasks, their ability to navigate the nuances of human emotion—emotion cognition—remains a complex frontier. Traditional affective computing focused primarily on surface-level sentiment classification (e.g., labeling text as "happy" or "sad"). However, true emotional intelligence requires generating resonant responses, maintaining consistent personas, and understanding the underlying mental states of others.

The authors identify three core challenges currently facing the field:

  1. Uniqueness: Emotions in text are abstract and require deep contextual understanding rather than surface-level pattern matching.
  2. Methodological Complexity: Moving beyond simple labels necessitates tailored strategies for both generative tasks and quantitative evaluation.
  3. Task Diversity: Emotional applications range from mental health support to humor detection, each requiring distinct success metrics.

A Unified Framework for Emotion Cognition

To address the lack of a cohesive structure in this sub-field, the paper organizes current research using Ulric Neisser’s cognitive psychology framework. This approach breaks LLM capabilities into seven distinct stages, ranging from basic data input to higher-order reflection.

Conceptual framework of emotion cognition in LLMs, mapping bases, methods, and future directions.

The Seven Stages of LLM Cognition

  • Sensation: The initial processing of input data via prompt engineering, embeddings, and knowledge enhancement.
  • Perception: Interpreting sensory data through emotion recognition and explainable AI (understanding why a model identifies a specific emotion).
  • Imagination: The creative generation of emotional content, including stories, poetry, and humor.
  • Retention: Encoding and storing emotional "memories" to build consistent character profiles or personas.
  • Recall: Retrieving relevant emotional history to maintain context in long-term interactions.
  • Problem-Solving: Applying emotional intelligence to downstream tasks like mental health counseling or adaptive education.
  • Thinking: Higher-order reflection, specifically focusing on Theory of Mind (ToM)—the ability to infer the mental states, beliefs, and intentions of others.

State-of-the-Art Performance and Benchmarks

The survey references a wide array of benchmarks used to evaluate these stages, such as IEMOCAP and MELD for perception, and ESConv for empathetic dialogue. Specialized models often outperform general-purpose LLMs in niche emotional tasks.

Summary of key research papers across the seven stages of emotion cognition.

Key performance highlights include:

  • Mental Health: Fine-tuned models like Mental-Alpaca outperform GPT-4 in mental health prediction by approximately 4.8% in balanced accuracy.
  • Emotion Recognition: The InstructERC framework, utilizing an emotion alignment task, currently holds state-of-the-art results on major conversational datasets.
  • Social Reasoning: GPT-4 mirrors human-like patterns in moral judgment, outperforming specialized models by 6.2% in F1 scores when using "Moral Chain-of-Thought" prompting.
  • Humor Generation: Systems like Witscript 3 can generate responses that human evaluators perceive as jokes 44% of the time.

Key Insights and the "Clever Hans" Effect

Despite these gains, the research reveals significant limitations. A primary concern is the "Clever Hans" effect in Theory of Mind: while GPT-4 shows human-like reasoning, its performance is brittle. Minor variations in task wording can cause accuracy to plummet from 99% to below 50%, suggesting models may rely on superficial heuristics rather than true cognitive "thinking."

Furthermore, LLMs face a "Humor Ceiling." While they are excellent at explaining why a joke is funny, they struggle with originality, often replicating existing jokes from their training data rather than generating novel humor. Interestingly, in healthcare settings, LLM-generated responses are often perceived as more empathetic than those from human professionals, even though the model lacks a genuine internal emotional state.

Future Research Directions

The authors identify several "open frontiers" for the research community:

  • Generalization and Bias: Current research is heavily skewed toward English-language datasets. There is a critical need for manually annotated data in diverse languages to avoid "emotional Westernization."
  • Explainability: While we can measure outputs, the internal "decision-making" process of how an LLM arrives at an emotional conclusion remains opaque.
  • Multimodality: Future models must integrate tone, facial expressions, and physiological cues to move beyond text-only emotion cognition.
  • Originality vs. Replication: Moving LLMs from "stochastic parrots" to truly creative emotional agents remains the primary challenge for the next generation of AI development.

Summary updated

Abstract

Emotion cognition in large language models (LLMs) is crucial for enhancing performance across various applications, such as social media, human-computer interaction, and mental health assessment. We explore the current landscape of research, which primarily revolves around emotion classification, emotionally rich response generation, and Theory of Mind assessments, while acknowledge the challenges like dependency on annotated data and complexity in emotion processing. In this paper, we present a detailed survey of recent progress in LLMs for emotion cognition. We explore key research studies, methodologies, outcomes, and resources, aligning them with Ulric Neisser's cognitive stages. Additionally, we outline potential future directions for research in this evolving field, including unsupervised learning approaches and the development of more complex and interpretable emotion cognition LLMs. We also discuss advanced methods such as contrastive learning used to improve LLMs' emotion cognition capabilities.
BibTeX
@misc{chen2024recent,
  author = {Yuyan Chen and Yanghua Xiao},
  title = {{Recent Advancement of Emotion Cognition in Large Language Models}},
  year = {2024},
  eprint = {2409.13354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.13354},
}
Figure