Enhancing Empathetic Response Generation by Augmenting LLMs with Small-scale Empathetic Models

Zhou Yang, Zhaochun Ren, Yufeng Wang, Shizhong Peng, Haizhou Sun, Xiaofei Zhu, Xiangwen LiaoarXiv 2024 · DOI · arXiv

The Capability Gap in Empathetic Dialogue

Large Language Models (LLMs) like ChatGPT have revolutionized conversational AI with their eloquence and fluency. However, they often exhibit "emotional bluntness," struggling to identify fine-grained emotions or pinpoint specific triggers (emotion causes) within a dialogue. While they excel at general conversation, the nuances of empathy—such as distinguishing "sentimental" from "sad"—remain a challenge.

In contrast, specialized Small-scale Empathetic Models (SEMs) are highly effective at emotion recognition and cause identification because they are trained on targeted datasets. Yet, their generative capabilities are often lackluster, resulting in repetitive or robotic responses. This research proposes the Hybrid Empathetic Framework (HEF) to bridge this gap, using the SEM as an analytical "brain" and the LLM as an expressive "voice."

Complementary capabilities of SEMs and LLMs in empathetic dialogue.

The Hybrid Empathetic Framework (HEF)

The HEF is a non-fine-tuning approach that augments LLMs through structured prompt engineering. By leveraging SEMs as specialized plugins, the framework guides LLMs to generate responses that are more empathetic and contextually relevant.

Overview of the three-step HEF process.

The process consists of three primary stages:

  1. Training the SEM: An improved version of the Emotion-Semantic Correlation Model (ESCMtt) is trained on empathetic datasets to become an expert in fine-grained emotion detection.
  2. Information Extraction:
    • Two-stage Emotion Prediction: The SEM identifies the top-$k$ most likely emotion categories from a set of 32.
  3. Emotion Cause Perception: Using the SEM’s internal attention weights, the framework identifies "high-weight" words—the specific parts of the user's input that triggered the emotion.
  4. Guided Generation: These insights are integrated into a structured prompt for the LLM. The LLM is instructed to prioritize the SEM's predicted emotions and focus on the identified "cause words" when formulating its response.

Key Insights and Results

The core innovation of HEF lies in narrowing the LLM's "search space." By providing the SEM's top picks, a complex 32-way classification task is simplified into a manageable choice for the LLM. Furthermore, using attention weights to guide the LLM's focus effectively transfers "understanding" across different model architectures.

Performance Gains

  • Accuracy: HEF significantly improves emotion prediction accuracy. For instance, ChatGPT’s accuracy rose from 37.9% to 45.63% when augmented with the framework.
  • Human-Centric Evaluation: In head-to-head comparisons using GPT-4 as an evaluator, the HEF-augmented ChatGPT outperformed the base version by a margin of 86 wins to 1 loss in terms of empathy.
  • Precision: While HEF-augmented models sometimes showed lower diversity scores, the authors argue this reflects higher precision and relevance rather than a lack of creativity.

Conclusion and Future Directions

The success of the HEF depends heavily on the LLM's ability to follow complex instructions. While models like ChatGPT and Mistral integrated SEM "hints" effectively, smaller models like Llama2-7b struggled. Future work will explore extending this "plugin" mechanism to other multi-classification tasks and refining the framework to be even more lightweight. By combining the analytical strengths of small models with the expressive power of large ones, HEF provides a scalable path toward more emotionally intelligent AI.

Summary updated

Abstract

Empathetic response generation is increasingly significant in AI, necessitating nuanced emotional and cognitive understanding coupled with articulate response expression. Current large language models (LLMs) excel in response expression; however, they lack the ability to deeply understand emotional and cognitive nuances, particularly in pinpointing fine-grained emotions and their triggers. Conversely, small-scale empathetic models (SEMs) offer strength in fine-grained emotion detection and detailed emotion cause identification. To harness the complementary strengths of both LLMs and SEMs, we introduce a Hybrid Empathetic Framework (HEF). HEF regards SEMs as flexible plugins to improve LLM's nuanced emotional and cognitive understanding. Regarding emotional understanding, HEF implements a two-stage emotion prediction strategy, encouraging LLMs to prioritize primary emotions emphasized by SEMs, followed by other categories, substantially alleviates the difficulties for LLMs in fine-grained emotion detection. Regarding cognitive understanding, HEF employs an emotion cause perception strategy, prompting LLMs to focus on crucial emotion-eliciting words identified by SEMs, thus boosting LLMs' capabilities in identifying emotion causes. This collaborative approach enables LLMs to discern emotions more precisely and formulate empathetic responses. We validate HEF on the Empathetic-Dialogue dataset, and the findings indicate that our framework enhances the refined understanding of LLMs and their ability to convey empathetic responses.
BibTeX
@misc{yang2024enhancing,
  author = {Zhou Yang and Zhaochun Ren and Yufeng Wang and Shizhong Peng and Haizhou Sun and Xiaofei Zhu and Xiangwen Liao},
  title = {{Enhancing Empathetic Response Generation by Augmenting LLMs with Small-scale Empathetic Models}},
  year = {2024},
  eprint = {2402.11801},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.11801},
}
Figure