The Emotional Spectrum of LLMs: Leveraging Empathy and Emotion-Based Markers for Mental Health Support

Alessandro De Grandi, Federico Ravenda, Andrea Raballo, Fabio CrestaniNAACL 2025

TL;DR

This paper introduces RACLETTE, a novel conversational AI system that leverages Large Language Models (LLMs) to understand and generate empathetic responses for mental health support. By creating explainable "emotion embeddings," RACLETTE can build user emotional profiles that may serve as markers for preliminary mental health assessment and diagnosis.

What Problem Does This Paper Address?

The demand for mental health services is growing, and there's a critical need for innovative, accessible solutions. Conversational AI, particularly empathetic chatbots, holds promise for providing support, but current systems often lack nuanced emotional understanding and the ability to generate truly personalized, supportive responses. Furthermore, the scarcity of sensitive mental health data makes training robust models challenging. This paper addresses the challenge of developing AI that can not only understand and respond empathetically to users' emotional states but also translate these emotional patterns into interpretable markers for mental health assessment, potentially aiding in early detection and diagnosis.

Methods

The core of this work lies in the development of RACLETTE, a system built upon the Mistral 7B LLM. Key methodological contributions include:

  • Empathetic Conversational Model: RACLETTE is fine-tuned using a novel 3-turn conversational structure (Prompt, Emotion, Response) and an unconventional approach to emotion classification. Instead of a single label, it predicts a distribution of emotions for each turn.
  • Emotion Embeddings: The system generates "emotion embeddings," which are high-dimensional vectors representing a user's emotional state. These embeddings are created by aggregating predicted emotion distributions over a conversation, allowing for a nuanced, multi-faceted emotional profile.
  • Explainable Markers: These emotion embeddings are designed to be interpretable. The authors propose that by comparing a user's aggregated emotion profile with pre-computed profiles associated with specific mental disorders (derived from specialized datasets), it's possible to identify potential mental health conditions.
  • Fine-tuning Strategy: The Mistral 7B model is fine-tuned using the Empathetic Dialogues Dataset, leveraging its autoregressive properties for both classification and response generation.

Datasets

  • Empathetic Dialogues Dataset (Rashkin et al., 2018): Used for training the RACLETTE model to identify emotions and respond empathetically. This dataset contains 24,850 one-to-one open-domain conversations and is noted for its wide range of emotions.
  • Reddit Mental Health Dataset (Low et al., 2020): A collection of posts from various Reddit forums (subreddits) used to construct discrete emotion distributions associated with specific mental disorders, forming the basis for emotion embeddings.
  • DailyDialog Dataset (Li et al., 2017): A collection of 13,118 dialogues used as a control group for establishing a baseline emotional profile.

Key Insights and Ideas

  • Emotions as Diagnostic Markers: The paper posits that mental states can be represented as collections of emotions, and these emotion embeddings can serve as interpretable markers for mental health assessment, potentially aiding in early diagnosis.
  • LLMs for Nuanced Emotion Understanding: Fine-tuning LLMs for emotion classification, even using a generative approach, can lead to superior accuracy compared to traditional methods, especially when considering the full conversational context.
  • Beyond Single Emotion Labels: The approach of generating emotion embeddings as mixtures of discrete emotions acknowledges the complexity of human emotional expression, moving beyond simplistic one-hot encoding.
  • Subtle Empathy Mimicry: The model can sometimes mimic the emotional tone of the user, reflecting a natural aspect of human empathy where emotions can be contagious.

Key Results

  • Emotion Classification Accuracy: RACLETTE achieved an overall emotional accuracy of 59% on conversations from the Empathetic Dialogues test set, outperforming several benchmarks (e.g., CAiRE at 51%, Chen et al. at 53%).
  • Empathetic Response Quality: The model achieved a BERTSCORE of 0.87, indicating high semantic similarity between its generated responses and human-provided target responses.
  • Mental Health Classification: In an experiment to detect suicide risk, RACLETTE's combined similarity metrics achieved a recall of 0.95, demonstrating a strong ability to identify relevant cases, albeit with a precision of 0.63. The KL Divergence variant showed a balanced performance with precision 0.71 and recall 0.90.
  • Explainable Embeddings: Visualizations of emotion embeddings for various mental health subreddits (e.g., depression, anxiety, PTSD) showed distinct patterns and similarities, aligning with psychological understanding of these conditions.

Limitations and Future Work

  • Data Quality and Reliability: The quality and accuracy of emotional data used for training are critical. Emotional data collection is complex and time-consuming, and self-reported information can be unreliable due to biases or misinterpretations.
  • Noise and Ambiguity: Online data can contain noise (e.g., users seeking advice for others) and intentional misreporting. Individual expression varies greatly, and the model's predictions can reflect uncertainty, especially with short or ambiguous prompts.
  • Confounding Factors: Comorbidities can lead to complex emotional profiles that are difficult to parse.
  • Future Work:
    • Extend RACLETTE's capabilities beyond purely supportive interactions to incorporate a broader range of therapeutic approaches.
  • Incorporate cognitive symptoms (memory, attention, perception) alongside affective components for a more holistic assessment.
  • Expand emotion embeddings to include markers of cognitive functioning for more comprehensive profiles.
  • Address ethical implications, including potential misuse, bias, and the need for safeguards.

Summary updated

BibTeX
@inproceedings{grandi2025emotional,
  author = {Alessandro De Grandi and Federico Ravenda and Andrea Raballo and Fabio Crestani},
  title = {{The Emotional Spectrum of LLMs: Leveraging Empathy and Emotion-Based Markers for Mental Health Support}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure