Would you like to tell me more? Generating a corpus of psychotherapy dialogues

Seyed Mahed Mousavi, Alessandra Cervone, Morena Danieli, Giuseppe RiccardiNAACL 2021

TL;DR

This paper introduces a novel methodology for generating a corpus of psychotherapy dialogues, specifically focusing on follow-up conversations in the context of Cognitive Behavioral Therapy (CBT). By leveraging personal narratives and an automated stimulus generation process, they create a valuable dataset for training more empathetic and effective conversational agents in the mental health domain.

What Problem Does This Paper Address?

Developing effective conversational agents for mental health support is a significant challenge. Current approaches often rely on rule-based systems or simple dialogue flows, leading to shallow interactions and limited user engagement. This is largely due to the complexity of mental health conversations, which involve personal feelings, user-specific situations, and nuanced emotional states. Existing dialogue corpora, often collected for tasks like information retrieval or slot-filling, are not well-suited for the deeply personal and longitudinal nature of mental health support. Specifically, there's a lack of data that captures the follow-up nature of therapeutic interventions, where agents need to recall past interactions and monitor patient progress over time.

Methods

The core of this paper is a multi-stage methodology for generating psychotherapy dialogues:

  1. Collection of Personal Narratives: Users receiving CBT were asked to write notes about daily events that triggered emotional arousal, using a mobile application that guided them through the ABC (Antecedent, Belief, Consequences) technique. This technique helps users understand how their thoughts influence their emotions.
  2. Stimulus Generation: An automatic algorithm was developed to create textual stimuli for dialogue writers. This algorithm extracts a key sentence from the personal narratives, incorporates the associated emotion (either user-provided or automatically detected via sentiment analysis), and uses predefined templates to generate a "common-ground statement" followed by a follow-up question. This mimics how psychotherapists might initiate a review of a patient's progress.
  3. Dialogue Generation: The generated stimuli were presented to two groups of dialogue writers: psychotherapists experienced in CBT and non-expert writers. Both groups were given detailed guidelines and examples to write follow-up dialogues, impersonating both the Personal Healthcare Agent (PHA) and the patient. The goal was to create dialogues that elaborated on the event introduced in the stimulus and included a closure turn.

Datasets

The primary dataset introduced in this paper is a corpus of follow-up psychotherapy dialogues. This corpus was generated using the described methodology and consists of 800 dialogues. The authors also mention using a lexicon-based sentiment analyzer from The OpeNER project for emotion detection and a summarizer from the sumy library for stimulus generation.

Key Insights and Ideas

  • The importance of longitudinal data: The paper highlights that mental health support requires dialogues that span over time, mirroring real therapeutic interventions, which is a significant gap in current dialogue corpora.
  • Automated stimulus generation bridges the gap: The proposed algorithm for generating stimuli from personal narratives is a key innovation, enabling the creation of contextually relevant prompts for dialogue writers.
  • Domain expertise matters, but guidelines can compensate: While psychotherapists showed more nuanced validation of stimuli, non-expert writers, with appropriate guidelines, could still produce a course-grained validation, suggesting a scalable approach to data collection.
  • Differences in dialogue writing: Experts tended to write shorter conversations but used a richer vocabulary, while non-experts produced more verbose dialogues with less lexical diversity. This suggests that domain expertise influences conversational style.
  • Dialogue Acts reveal structural differences: Professionals used a more structured set of dialogue acts, focusing on core therapeutic functions, whereas non-experts incorporated more varied, sometimes less relevant, dialogue acts.

Key Results

  • Corpus Size: 800 follow-up dialogues were collected.
  • Stimuli Validation: Psychotherapists had a higher validation rate for stimuli (44.5% invalid) compared to non-experts (34.2% invalid), indicating domain expertise leads to more precise assessment.
  • Dialogue Length: Experts wrote shorter dialogues (average 3.7 turns) than non-experts (average 4.2 turns).
  • Vocabulary: Experts used a wider range of unique tokens (4251) compared to non-experts (3146).
  • Response-Selection Model Evaluation: A baseline TF-IDF response-selection model trained on this corpus achieved promising results. Human evaluation showed that 91% of system turns were considered appropriate and coherent, leading to 72% of dialogues being rated as acceptable quality.

Limitations and Future Work

  • Anonymization: The authors note that they are seeking further funding to anonymize the corpus for public release, respecting patient privacy.
  • Scalability of Expert Writers: While psychotherapists are valuable, their availability is limited. The paper suggests that training a larger number of non-expert writers with appropriate guidelines could be a more cost-effective and scalable approach for data collection.
  • Further Model Development: The current work focuses on generating the corpus and evaluating a baseline response-selection model. Future work could involve training more sophisticated dialogue models, including end-to-end systems, using this dataset.
  • Broader Applicability: The methodology could be extended to other domains requiring personal, longitudinal conversations.

Summary updated

BibTeX
@inproceedings{mousavi2021would,
  author = {Seyed Mahed Mousavi and Alessandra Cervone and Morena Danieli and Giuseppe Riccardi},
  title = {{Would you like to tell me more? Generating a corpus of psychotherapy dialogues}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure