Abstract
Empathy is critical for effective and satisfactory conversational communication. Prior efforts to measure conversational empathy mostly focus on expressed communicative intents—that is, the way empathy is expressed. Yet, these works ignore the fact that conversation is also a collaboration involving both speakers and listeners. In contrast, we propose a multi-dimensional empathy evaluation framework to measure both expressed intents from the speaker’s perspective and perceived empathy from the listener’s perspective. We apply our analytical framework to examine internal customer-service dialogues. We find the two dimensions (expressed intent types and perceived empathy) are interconnected, while perceived empathy has high correlations with dialogue satisfaction levels. To reduce the annotation cost, we explore different options to automatically measure conversational empathy: prompting LLMs and training language model-based classifiers. Our experiments show that prompting methods with even popular models like GPT-4 and Flan family models perform relatively poorly on both public and our internal datasets. In contrast, instruction-finetuned classifiers based on FlanT5 family models outperform prior works and competitive baselines. We conduct a detailed ablation study to give more insights into instruction finetuning method’s strong performance.
Introduction
Empathy in conversation is a two-way street, moving beyond just what a speaker expresses to how a listener perceives it. Most research on "computational empathy" treats it as a one-sided act, focusing on whether an AI agent uses the right words to express concern. This "individualistic" view ignores the relational nature of conversation. For instance, a therapist's clarification might be empathetic, while the same question from a customer service agent might feel cold or intrusive.
We identify two major gaps in the current landscape: the lack of a listener's perspective in existing benchmarks and the narrow focus on mental health support. This paper addresses these by introducing a multi-dimensional evaluation framework and demonstrating that while large language models (LLMs) like GPT-4 struggle with zero-shot empathy detection, instruction-finetuned models achieve state-of-the-art performance.
A Multi-dimensional Framework
We propose a two-dimensional framework to capture the complexity of empathetic exchange:
- Expressed Empathy: Categorizing 16 specific communicative intents (e.g., "aware of problem," "express reassurance," "tentative solution").
- Perceived Empathy: Measuring four fine-grained aspects from the listener's perspective: Engagement, Understanding, Sympathy, and Helpfulness.
To validate this framework, we utilized three datasets:
- EMH (Empathy Mental Health): A public dataset of single-round mental health support dialogues.
- ESConv (Emotional Support Conversation): A public dataset of multi-round peer-to-peer support.
- Empeval (Internal): A newly annotated dataset of 2,000 real-world customer service sessions. This moves empathy research into a commercial, task-oriented context where empathy is a key driver of user satisfaction. Each utterance was judged by human raters on the 16 intents and 4 perceived dimensions using Likert scales.
Experimental Results
We compared three modeling approaches: standard encoder models (BERT, RoBERTa), prompting LLMs (GPT-4, Flan-T5-XXL, Flan-UL2), and instruction-finetuned classifiers (Flan-T5 Large and XL).
Key Findings
- Prompting is Not Enough: Even "frontier" models like GPT-4 perform relatively poorly at zero-shot empathy detection. On the EMH dataset, GPT-4 (Zero-shot) only achieved a 45.4% Macro F1, trailing far behind fine-tuned models.
- Instruction-Finetuning Excellence: Instruction-finetuned Flan-T5-XL achieved the best results across almost all metrics. Adding natural language instructions during fine-tuning (e.g., providing a definition of "Emotional Reaction") significantly boosted performance, increasing Macro F1 from 72.4% to 78.0% on the internal dataset.
- Empathy Drives Satisfaction: Perceived empathy dimensions, especially Helpfulness and Understanding, showed high Spearman correlations with overall dialogue satisfaction. If the customer feels understood, they are significantly more likely to be satisfied.
- Context and Scale: Performance gains largely plateaued at the "Large" model size (770M parameters). Interestingly, the 3B parameter model (Flan-T5-XL) showed signs of overfitting on specific labels, suggesting that massive scale isn't always better than high-quality, task-specific fine-tuning.
Conclusion
This work highlights that empathy is a dynamic process where the listener's reaction is often the best evidence of a speaker's empathy. While the internal "Empeval" dataset cannot be released due to privacy, the framework and findings provide a roadmap for evaluating empathy in task-oriented AI. Future work will explore applying this framework to real-time human-AI interactions to evaluate how humans perceive empathy from automated agents.
Summary updated