Towards augmenting crisis counselor training by improving message retrieval

Orianna Demasi, Marti A. Hearst, Benjamin RechtNAACL 2019

TL;DR

This paper introduces a novel approach to training crisis counselors by simulating distressed individuals in a low-risk environment. They leverage recent advancements in text embeddings to improve message retrieval for building a conversational AI, demonstrating that retrieval can be a competitive alternative to generative models in data-limited scenarios.

What Problem Does This Paper Address?

Training crisis counselors is challenging because it requires realistic practice without endangering individuals in distress. Traditional role-playing is time-consuming and resource-intensive. While automated systems could scale training, replacing human empathy with AI is not the goal. Instead, this paper aims to augment training by creating a simulated distressed individual that novice counselors can interact with, allowing them to practice their skills in a safe, low-risk setting. A key challenge in building such a system, especially with limited data, is generating coherent and engaging responses.

Methods

The authors explore two main tasks: retrieving a visitor's response to a single counselor message and retrieving a response considering conversational context.

For single-message retrieval, they compare:

  • TF-IDF: A traditional method for text representation.
  • Embedding-based retrieval: Utilizing various pre-trained word and sentence embeddings (GloVe, FastText, ELMo, InferSent, GenSen) to calculate cosine similarity between messages.
  • Seq2Seq-based retrieval (S2S-retrieve): Using a sequence-to-sequence model to predict the likelihood of a response given a counselor's message.

To incorporate conversational context (specifically, one preceding visitor message), they extend the retrieval methods by:

  • Summing embeddings: Calculating similarity over the sum of embeddings of the current message and the preceding context.
  • Weighted sum: Combining message similarity and context similarity with a learned weight.
  • Concatenating context: For Seq2Seq models, preceding messages are concatenated before encoding.

They also consider generative models (Seq2Seq and a hierarchical RNN) as a baseline for comparison.

Datasets

The authors collected a novel dataset of synthetic chat transcripts for suicide prevention counselors and hotline visitors. These transcripts were generated by experienced counselors role-playing both sides of a conversation to protect the privacy of real individuals. The dataset was collected in three phases:

  1. Full role-plays: Counselors role-played both the counselor and visitor in a simulated crisis text conversation.
  2. Visitor-only role-plays: Counselors role-played only the visitor, responding to a given counselor transcript.
  3. Counselor paraphrases: Counselors reworded or improved existing counselor messages from full role-plays.

The dataset consists of 32 crisis counselors and includes unique conversations, visitor-only and counselor-only role-plays, and messages/paraphrases.

Key Insights and Ideas

  • Retrieval as a viable alternative: In data-limited settings, retrieval methods, especially those powered by modern embeddings, can be competitive with or even surpass generative models for generating coherent responses.
  • The power of embeddings: Recent embedding techniques significantly improve retrieval performance compared to traditional TF-IDF, suggesting that advancements in text representation are crucial for dialogue systems.
  • Context matters, but complicates generation: While incorporating conversational context generally improves retrieval, it poses a greater challenge for generative models, potentially leading to less engaging responses.
  • Evaluation is key: Human evaluation is essential for assessing response quality, and the authors highlight the importance of clear instructions and multiple annotators to ensure reliable judgments.

Key Results

  • Single Message Retrieval:
    • Retrieval methods significantly outperformed random selection.
  • ELMo embeddings, along with FastText, InferSent, and GenSen, improved upon the TF-IDF baseline.
  • ELMo embeddings were the only retrieval method to surpass the Seq2Seq generative model in terms of coherent messages.
  • S2S-retrieve (Seq2Seq likelihood) achieved the highest percentage of coherent responses but produced the shortest messages.
  • Contextual Retrieval (one preceding message):
    • Ground truth responses were judged as more coherent when context was provided.
  • ELMo-based retrieval saw a slight decrease in coherence when context was added, while S2S-retrieve maintained a high coherence rate.
  • Generative models struggled with context, with retrieval methods generally outperforming them.

Limitations and Future Work

  • Data limitations: The study was conducted in a data-limited setting, which may have impacted the performance of generative models. Extensive hyper-parameter tuning could potentially improve results.
  • Limited context: The current work only considers one preceding message of context. Given the length of the collected conversations, more extensive context is likely necessary for truly coherent simulations.
  • Future directions: The authors plan to explore leveraging richer conversational structure and the counselor's protocol. They also suggest investigating hybrid retrieval-generation approaches, where retrieved prototypes are modified to tailor responses to similar contexts.

Summary updated

BibTeX
@inproceedings{demasi2019towards,
  author = {Orianna Demasi and Marti A. Hearst and Benjamin Recht},
  title = {{Towards augmenting crisis counselor training by improving message retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure