TL;DR
This paper introduces MultiAgentESC, a novel training-free framework that uses a multi-agent system to emulate human-like emotional support conversations. By breaking down the process into dialogue analysis, strategy deliberation, and response generation, MultiAgentESC aims to overcome LLM biases and improve the diversity and effectiveness of emotional support.
What Problem Does This Paper Address?
The paper addresses the critical need for effective emotional support conversations (ESC) systems, especially given the growing mental health pressures and the shortage of human psychologists. Existing ESC systems often focus on direct response generation and overlook the importance of domain-specific reasoning and expert collaboration. Furthermore, current methods struggle with the "one-to-many" problem of strategy selection (where multiple strategies can be effective for a given situation) and are prone to LLM biases that favor certain strategies over others. This limits their ability to provide nuanced and truly helpful emotional support.
Methods
MultiAgentESC employs a three-stage, multi-agent framework designed to mimic how human experts provide emotional support:
- Dialogue Analysis: Specialized agents analyze the conversation to extract the user's emotional state, the specific event causing it, and their underlying intention. This stage can be bypassed for simpler conversations.
- Strategy Deliberation: To combat LLM bias and the one-to-many strategy problem, this stage incorporates similar past cases (retrieved using SBERT) as experience. Agents deliberate to identify and retain multiple valid support strategies, encouraging diversity.
- Response Generation: Multiple responses are generated using the retained strategies. These responses then undergo a multi-agent debate and refinement process to select the most optimal and empathetic response.
The framework is implemented using AutoGen, an open-source framework for building AI agents and facilitating multi-agent collaboration.
Datasets
The experiments were conducted on the ESConv dataset, which contains approximately 1,000 conversations and 31,000 utterances related to emotional support. The dataset was collected in a help-seeker and supporter mode and categorizes supporter responses into eight distinct types. For evaluation, 100 conversations were randomly selected for the test set, while the remainder were used for extracting analogous cases.
Key Insights and Ideas
- Emulating Human Expertise: The core idea is to move beyond direct response generation and simulate the multi-stage, collaborative decision-making process of human psychological experts.
- Mitigating LLM Bias: By incorporating past experiences and facilitating multi-agent deliberation, the framework actively works to reduce the LLM's inherent preference for certain strategies.
- Addressing the "One-to-Many" Strategy Problem: Instead of forcing a single optimal strategy, the system generates multiple responses from diverse strategies and then selects the best one through collaboration.
- Response-Centered Approach: The final selection of the response is prioritized over pre-determined strategies, allowing for more flexibility and better alignment with the user's needs.
Key Results
- Automatic Evaluation: MultiAgentESC achieved superior results on diversity metrics (D-1 and D-2) and strong performance on quality metrics (B-n, F1, R-L) across both LLaMA3-70b and Qwen2.5-32b models. This highlights its ability to generate diverse and high-quality emotional support responses.
- Strategy Distribution: The framework demonstrated a more balanced and diversified distribution of strategy selection compared to baseline methods, with all eight strategies showing utilization rates exceeding 3% and none dominating.
- Human Evaluation: Human annotators rated MultiAgentESC as superior across all aspects, including Fluency, Identification, Comforting, Suggestion, and Overall effectiveness, indicating its strong ability to provide empathetic and actionable support.
- GPT-40 Judgements: Evaluation using GPT-40 corroborated human findings, showing MultiAgentESC's strengths in Identification and Suggestion.
- Ablation Study: Removing components like dialogue analysis, experience, or group discussion led to a drop in performance, underscoring the importance of each part of the framework. The "w/o experience" variant showed significant deviation, highlighting the role of prior knowledge in reducing bias.
Limitations and Future Work
- Distinguishing Analogous Strategies: The framework still faces challenges in differentiating between strategies that serve similar functions, which can lead to a preference for more commonly used ones.
- Persistent Strategy Bias: While mitigated, the bias in strategy utilization remains a challenge, suggesting the need for more advanced strategy selection methods.
- Future Work: The authors plan to extend MultiAgentESC to multimodal emotional support dialogue systems, incorporating multimodal LLMs to understand emotions from various sources. They also aim to improve its ability to generalize across different domains.
Summary updated