Abstract
As Large Language Models (LLMs) achieve human-level proficiency in logic and reasoning, their capacity for emotional intelligence remains a critical frontier. This paper introduces EmotionQueen, a rigorous benchmark designed to evaluate LLMs across complex, real-world emotional scenarios. By assessing models on four distinct tasks—Key Event Recognition, Mixed Event Recognition, Implicit Emotional Recognition, and Intention Recognition—we reveal a significant "empathy gap": a model’s ability to correctly identify an emotion (PASS rate) has almost no correlation with its ability to generate an empathetic response (WIN rate). Our findings highlight the nuances of machine empathy and identify top-performing models like LLaMA2-70B and Claude2.
1. Introduction
While LLMs have become incredibly proficient at technical tasks like coding and mathematical reasoning, they often remain "emotionally numb" in conversational contexts. Traditional evaluations of emotional intelligence in AI typically focus on "atomic" tasks, such as simple sentiment analysis (e.g., classifying a sentence as happy or sad). However, human empathy is far more complex, requiring the navigation of mixed emotions, implicit feelings, and the underlying intentions behind a user's communication.
To address this, we present EmotionQueen, a unified, objective, and multi-faceted benchmark that distinguishes between a model's ability to "detect" an emotion and its ability to actually "empathize" with a human user.
2. The EmotionQueen Framework
The EmotionQueen framework evaluates models across four sophisticated tasks designed to mirror real-world human interactions:
- Key Event Recognition (KER): Identifying the most emotionally significant event in a statement that contains both routine and significant information.
- Mixed Event Recognition (MER): Responding to two events of similar importance simultaneously (e.g., balancing a professional achievement with a personal setback).
- Implicit Emotional Recognition (IER): Identifying deep, underlying emotions that are not explicitly stated in the text.
- Intention Recognition (IR): Understanding the user's underlying purpose or need to provide specific, helpful solutions.
2.1 Evaluation Metrics
To quantify performance, we utilize two primary metrics:
- PASS Rate: Measures the model's accuracy in identifying the correct event or emotion (the "understanding" phase).
- WIN Rate: Measures the model's ability to generate a response that is genuinely empathetic, helpful, or supportive (the "responding" phase).
Evaluation was conducted using GPT-4 as an automated judge. This approach demonstrated a remarkably high correlation (0.991) with human annotators, specifically female psychology graduate students, ensuring the benchmark's reliability.
3. Dataset Construction
The researchers constructed a massive dataset of 10,000 user statements generated via GPT-4, covering five primary life domains: Achievements, Family and Friends, Health Status, Economic Status, and Accidents.
The dataset maintains a 3:7 ratio of positive to negative statements, reflecting the reality that negative emotions typically require more complex and nuanced empathetic responses. To ensure the tasks mirrored real-world challenges, the authors benchmarked the models against existing public datasets, including MELD, EmotionLines, and DailyDialog.
4. Key Insights and Results
4.1 The "Empathy Gap"
The most striking finding from the EmotionQueen evaluation is that PASS and WIN rates are almost entirely uncorrelated. A model can be nearly perfect at identifying what happened (high PASS) but fail significantly at responding with appropriate empathy (low WIN). This suggests that emotional understanding and emotional response are two distinct capabilities within LLMs that must be developed independently.
4.2 The Challenge of Mixed Events
Mixed Event Recognition (MER) proved to be the "boss level" for most models. LLMs frequently struggle when presented with multiple events of equal weight, often hyper-focusing on one aspect while ignoring the other. This leads to responses that feel perfunctory or dismissive of the user's full context.
4.3 Model Performance
- Top Performers: LLaMA2-70B and Claude2 emerged as the leaders in emotional intelligence. LLaMA2-70B excelled in economic and accident-related scenarios, while Claude2 showed superior performance in achievements, health, and family/friends.
- Claude2’s Nuance: Claude2 was particularly adept at Mixed Event Recognition, achieving a 90.6% average score, demonstrating a unique ability to balance multiple emotional threads.
- Scaling Effects: Model size is a significant factor. Smaller models (e.g., BLOOM-7B or Falcon-7B) performed poorly, often failing to recognize emotional context entirely and providing generic, "robotic" replies.
4.4 AI vs. Human Empathy
Interestingly, in certain scenarios, top-tier models like LLaMA2-70B and Claude2 surpassed human-written responses in empathy scores. This suggests that with proper prompting and scale, AI can provide "idealized" emotional support that exceeds the average human interaction.
5. Limitations and Future Work
Despite the progress represented by EmotionQueen, several limitations remain:
- Subjectivity: Emotional interpretation is inherently subjective, and personal biases can influence even expert evaluations.
- Safety and Ethics: The benchmark does not currently measure the ethical safety of the advice provided. A model could theoretically be empathetic while suggesting harmful actions.
- Modality: The study is limited to text. Real-world empathy involves multimodal cues—tone of voice, facial expressions, and body language—which are not yet captured in this framework.
Future work will focus on expanding the framework to include a broader spectrum of emotional intelligence facets and developing more robust, objective metrics for ethical evaluation.
Summary updated