Large Language Models Produce Responses Perceived to be Empathic

Yoon Kyung Lee, Jina Suh, Hongli Zhan, Junyi Jessy Li, Desmond C. Ong

TL;DR

Recent research shows that Large Language Models (LLMs) like GPT-4 and Llama 2 generate responses to everyday personal struggles that human raters perceive as significantly more empathic and appropriate than those written by humans. While these models don't "feel" emotions, their ability to simulate supportive language suggests a massive potential for augmenting human peer support in mental health and social contexts.

What Problem Does This Paper Address?

Empathy is a cornerstone of social support, yet it is a "scarce resource." Human supporters—whether they are friends, peer counselors, or physicians—often face "empathy fatigue" or are simply too overburdened to provide the level of emotional validation a person needs.

While previous studies showed that AI could outperform doctors in providing empathic medical advice on Reddit, this paper addresses a broader gap: everyday social contexts. The authors investigate whether LLMs can provide contextualized empathy across diverse, non-medical scenarios like workplace conflict, parenting stress, and relationship issues. They also explore whether different AI models have distinct "personalities" or styles when trying to be helpful.

Methods

The researchers conducted two studies to evaluate AI-generated empathy:

  1. Study 1 (Human vs. AI): They selected 15 posts from Reddit (r/Parenting, r/relationships, r/workplace). They compared responses from a human research assistant (trained in psychology) against three LLMs: GPT-4 Turbo, Llama 2 (13B), and Mistral (7B).
  2. Study 2 (Generalizability): They expanded the dataset to 120 posts across six domains (adding Anger, Anxiety, and COVID-19 support) to see if the AI's performance held up across a wider variety of stressors.

Evaluation: Human raters (recruited via Prolific) performed blind ratings of the responses on a 5-point Likert scale for both "appropriateness" and "empathy."

Linguistic Analysis: The team used LIWC-22 (Linguistic Inquiry and Word Count) and bag-of-words models to identify the specific linguistic "fingerprints" of each AI model.

Datasets

  • Reddit Posts: 120 posts sourced from subreddits including r/Parenting, r/relationships, r/workplace, r/Anger, r/Anxiety, and r/COVID-19_support.
  • Newly Introduced: The authors have released their dataset of posts and multi-model empathic responses on GitHub.

Key Insights and Ideas

  • The Simulation Beats the Real Thing: Across the board, LLMs were rated as more empathic than the human baseline. This suggests that LLMs have effectively "distilled" the linguistic patterns of high-quality support from their massive training data.
  • Model "Personalities": The models aren't just generic text-bots; they have distinct styles:
  • Llama 2 is the "Casual Peer": It uses more emojis, exclamation marks, and informal language ("hey", "totally").
  • Mistral is the "Sympathetic Mourner": It relies heavily on apologies and condolences ("sorry to hear").
  • GPT-4 is the "Professional Supporter": It is structured, uses less "I" language, and focuses on validating the user's feelings with phrases like "completely understandable."
  • The Emoji Factor: In the regression analysis, the use of emojis (especially the "broken heart" 💔) was one of the strongest predictors of a response being rated as highly empathic.
  • Default Empathy: Prompting GPT-4 to be "extra empathic" didn't significantly raise its scores compared to a standard "peer supporter" prompt. This suggests that modern LLMs may already be operating near the "perceived empathy ceiling" for text-based responses.

Key Results

  • Empathy Ratings: In Study 1, human responses averaged 3.41/5. GPT-4 and Llama 2 both significantly outperformed the human, scoring 4.09/5. Mistral followed at 3.70/5.
  • Classification Accuracy: The researchers trained a logistic regression model that could identify which LLM wrote a response with startling accuracy (F1 scores: 0.92 for GPT-4, 0.96 for Llama 2, and 0.86 for Mistral). This proves that these models have highly predictable and distinct "voices."
  • Length Bias: LLM responses tended to be longer than human ones. While the researchers tried to control for this in prompts, the models' tendency toward verbosity may actually contribute to the perception of "effort" and, by extension, empathy.

Limitations and Future Work

  • Single-Turn Interaction: The study only looked at a single response to a single post. Real-world empathy is a "multi-turn" process that requires long-term memory and the ability to track a user's emotional state over time.
  • The "Deception" Problem: The authors acknowledge a major ethical hurdle: LLMs do not actually feel anything. There is an inherent deception in an AI saying "I understand how you feel," which could lead to a backlash if users feel manipulated.
  • Lack of Non-Linguistic Cues: Text-only empathy misses out on tone of voice, facial expressions, and physical presence, which are vital in high-stakes mental health crises.
  • Future Direction: The authors suggest exploring how to tailor AI "empathic styles" to individual preferences—some users might want the casual, emoji-heavy style of Llama 2, while others might prefer the structured, professional tone of GPT-4.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated surprising performance on many tasks, including writing supportive messages that display empathy. Here, we had these models generate empathic messages in response to posts describing common life experiences, such as workplace situations, parenting, relationships, and other anxiety- and anger-eliciting situations. Across two studies (N=192, 202), we showed human raters a variety of responses written by several models (GPT4 Turbo, Llama2, and Mistral), and had people rate these responses on how empathic they seemed to be. We found that LLM-generated responses were consistently rated as more empathic than human-written responses. Linguistic analyses also show that these models write in distinct, predictable ``styles", in terms of their use of punctuation, emojis, and certain words. These results highlight the potential of using LLMs to enhance human peer support in contexts where empathy is important.
BibTeX
@inproceedings{lee2024large,
  author = {Yoon Kyung Lee and Jina Suh and Hongli Zhan and Junyi Jessy Li and Desmond C. Ong},
  title = {{Large Language Models Produce Responses Perceived to be Empathic}},
  booktitle = {ACII},
  year = {2024},
  url = {https://openreview.net/forum?id=XjoG2mqmZH},
}
Figure