Steering Conversational Large Language Models for Long Emotional Support Conversations

Navid Madani, Sougata Saha, Rohini K. SrihariarXiv 2024 · arXiv · OpenReview

TL;DR

This paper tackles the challenge of making Large Language Models (LLMs) consistently follow emotional support strategies in long conversations. They introduce a new metric, Strategy-Relevant Attention (SRA), to measure how well LLMs focus on these strategies, and show that it correlates well with actual adherence. Their work also includes a new dataset and fine-tuned models that significantly improve strategy following.

What Problem Does This Paper Address?

Conversational AI is increasingly being developed for emotionally sensitive tasks like mental health support. A key challenge is ensuring that these AI models, particularly Large Language Models (LLMs), can consistently apply specific emotional support strategies (e.g., offering hope, validating emotions) throughout extended dialogues. Existing methods often focus on matching responses to gold-standard examples, which doesn't fully capture the nuanced nature of emotional support where multiple valid responses might exist for a given strategy. The authors aim to address this by developing a method to quantify and improve an LLM's ability to adhere to a prompted strategy over many turns, rather than just generating a plausible response.

Methods

The core of the paper introduces the Strategy-Relevant Attention (SRA) metric. This metric quantifies how much a model's attention is focused on the "strategy-relevant tokens" within the input prompt when generating a response. The hypothesis is that higher attention to these tokens leads to better strategy adherence. SRA is calculated by aggregating attention weights across all heads and layers for tokens identified as relevant to the strategy.

To facilitate their research, they also:

  • Created an Extended ESConv Dataset: This dataset is an expansion of the existing ESConv dataset, featuring synthetic strategy-conditioned continuations generated using Llama models and various prompting strategies.
  • Proposed Prompting Baselines: They experimented with different prompt templates to see how strategy guidelines' placement and the amount of conversation history affect adherence.
  • Fine-tuned Models: They fine-tuned Llama2 and Llama3 models using LoRA (Low-Rank Adaptation) on their extended dataset to improve their steerability and strategy adherence.

Datasets

  • ESConv Dataset: The original Emotional Support Conversation dataset, used as a base.
  • Extended ESConv Dataset: A new, synthetic dataset created by the authors. This dataset expands on ESConv by generating multi-turn, strategy-conditioned continuations. It was created by cutting conversations from the original ESConv dataset and prompting Llama models to generate responses following specific strategies.

Key Insights and Ideas

  • Attention is Key: The core insight is that an LLM's ability to follow a strategy in long conversations is strongly linked to where its attention is focused within the prompt. The SRA metric effectively captures this.
  • SRA Correlates with Adherence: The paper demonstrates a strong correlation between the SRA metric and actual strategy adherence, validated by both automated classifiers and human evaluations. This suggests SRA is a reliable proxy for steerability.
  • Prompt Engineering Matters: The placement of strategy guidelines within prompts and the amount of conversation history provided significantly impact a model's ability to adhere to strategies.
  • Fine-tuning Boosts Steerability: Fine-tuning LLMs on strategy-conditioned data, even with standard prompting, significantly improves their ability to maintain strategy adherence over long conversations.
  • Llama3 Shows Improvement but Still Struggles: While Llama3 models show better steerability than Llama2, they still exhibit a lack of consistent adherence to prompted strategies in extended dialogues, highlighting the ongoing challenge.

Key Results

  • SRA-Accuracy Correlation: A Pearson correlation of 0.94 was observed between SRA and strategy adherence accuracy for Llama2-7b-chat, and 0.8 for Llama3-8b-instruct.
  • Fine-tuned Model Performance: The authors' fine-tuned models showed significant improvements in strategy adherence, with an average improvement of 78.9% over Llama2 base models and 37.6% over Llama3 base models.
  • Prompt Template Performance: The cl_hf prompt template (strategy guidelines at the beginning, last 1 user/assistant message) achieved the most stable strategy following behavior among the tested baselines.
  • Human Evaluation Agreement: Human annotators' scores showed a high correlation (0.80 and 0.82) with the difference in SRA between responses, further validating SRA.
  • Model-Based Evaluation: Fine-tuned models significantly outperformed their base models in head-to-head comparisons, winning 74.12% of the time for Llama2 and 50.78% for Llama3.

Limitations and Future Work

  • Generalizability of SRA: The authors acknowledge that the generalizability of the SRA metric across diverse LLM architectures and configurations needs further exploration. Its effectiveness in scenarios beyond emotional support also requires investigation.
  • Focus on Adherence, Not Helpfulness: This study primarily focuses on strategy adherence. It does not directly measure the helpfulness or effectiveness of different strategies, leaving that for future work.
  • Ethical Considerations: The paper highlights the importance of user privacy, data anonymization, and ensuring empathetic, non-triggering responses in emotionally sensitive conversations. Continuous monitoring and human-in-the-loop oversight are crucial.
  • Future Work: The authors suggest exploring the planning aspect of emotional support tasks, investigating the helpfulness of different strategies, and further validating SRA in broader contexts. They also plan to refine long-horizon conversational planning.

Summary updated

Abstract

In this study, we address the challenge of consistently following emotional support strategies in long conversations by large language models (LLMs). We introduce the Strategy-Relevant Attention (SRA) metric, a model-agnostic measure designed to evaluate the effectiveness of LLMs in adhering to strategic prompts in emotional support contexts. By analyzing conversations within the Emotional Support Conversations dataset (ESConv) using LLaMA models, we demonstrate that SRA is significantly correlated with a model's ability to sustain the outlined strategy throughout the interactions. Our findings reveal that the application of SRA-informed prompts leads to enhanced strategic adherence, resulting in conversations that more reliably exhibit the desired emotional support strategies over longer conversations. Furthermore, we contribute a comprehensive, multi-branch synthetic conversation dataset for ESConv, featuring a variety of strategy continuations informed by our optimized prompting method. The code and data are publicly available on our Github.
BibTeX
@misc{madani2024steering,
  author = {Navid Madani and Sougata Saha and Rohini K. Srihari},
  title = {{Steering Conversational Large Language Models for Long Emotional Support Conversations}},
  year = {2024},
  eprint = {2402.10453},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.10453},
}
Figure