Identifying therapist conversational actions across diverse psychotherapeutic approaches

Fei-Tzin Lee, Derrick Hull, Jacob Levine, Bonnie K. Ray, Kathy McKeown2019 · DOI

TL;DR

This paper tackles the challenge of automatically understanding the conversational strategies therapists use across various therapeutic approaches. They introduce a novel, broad annotation scheme for therapist dialogue acts and demonstrate that a well-featured SVM classifier can effectively identify these actions, even outperforming more complex neural network models on their dataset. This work paves the way for automated analysis of therapeutic conversations, potentially leading to personalized recommendations for therapists.

What Problem Does This Paper Address?

Mental health treatment relies heavily on the dynamic interaction between therapist and client. Understanding the specific conversational "actions" a therapist takes (e.g., reflecting, problem-solving, empathizing) is crucial for grasping how therapeutic change occurs. However, manually analyzing vast amounts of therapy transcripts is prohibitively labor-intensive. Existing work often focuses on highly structured therapies like Motivational Interviewing or CBT. This paper aims to bridge this gap by developing a method to classify therapist dialogue acts across a diverse range of psychotherapeutic approaches, acknowledging the increased complexity and heterogeneity this brings. The ultimate goal is to enable automated analysis of therapeutic discourse to uncover patterns associated with clinical outcomes and inform the development of supportive tools for therapists.

Methods

The core of this work involves developing a novel annotation scheme for therapist dialogue acts. This scheme combines general dialogue act categories (like agreement, question) with clinically relevant codes derived from various therapeutic modalities (Emotion Focused Therapy, CBT, Motivational Interviewing, etc.). This broad approach is designed to capture therapist behaviors common across different therapeutic styles.

For classification, they experimented with two main approaches:

  • SVM with discrete features: This model leverages a rich set of features including sentence n-grams, dialogue information (speaker turn, utterance position), and crucially, context features (labels and n-grams from preceding utterances).
  • Neural network models: They explored a feedforward network on discrete features and a convolutional neural network (CNN) that processes both the text of the sentence and discrete features.

They found that the context of the sentence was a highly informative feature, and interestingly, using category-level labels as context outperformed using more granular act labels.

Datasets

The primary dataset used is a selection of psychotherapy transcripts from the Alexander Street Press corpus. Out of approximately 4,000 transcripts, they selected and annotated 340 transcripts. This resulted in 8,420 labeled client sentences and 9,056 labeled therapist sentences. For this work, they focused exclusively on classifying therapist acts. They also mention attempting to leverage the Switchboard corpus for transfer learning, but found it to be largely uninformative for their therapy-specific task, suggesting a significant difference between general conversation and therapeutic dialogue.

Key Insights and Ideas

  • Context is King: The most significant signal for identifying a therapist's dialogue act comes from the surrounding conversation. This highlights the importance of understanding the flow and history of the interaction.
  • SVM's Enduring Power: Surprisingly, a well-featured Support Vector Machine (SVM) classifier outperformed more complex neural network models on their dataset. This suggests that for this specific task and dataset size, carefully engineered features can be more effective than deep learning alone.
  • Broad Annotation is Feasible: Developing a unified annotation scheme that spans diverse therapeutic approaches is achievable and provides valuable insights into common therapeutic strategies.
  • Annotation Noise is a Challenge: The authors encountered instances where their model's predictions suggested potential errors in the original annotations, hinting at the utility of such models as annotation-checking tools.
  • Therapy Style Matters: Performance varied across different therapy styles, with longer sentences and more complex structures in some modalities posing greater challenges for classification.

Key Results

  • The SVM model with full feature sets achieved the best overall performance, measured by macro-F1 score.
  • The SVM achieved an accuracy of 74.98% and an F1 score of 69.94% on the test set categories.
  • The SVM significantly outperformed the baseline CNN and was also better than the feedforward neural network and a CNN combined with discrete features.
  • Context labels were identified as the most impactful feature, followed by sentence n-grams, in ablation studies.
  • Inter-annotator agreement for therapist acts was high (Cohen's kappa of 0.7900), indicating good reliability in the annotation process.

Limitations and Future Work

  • Data Scarcity: The primary limitation is the relatively small size of the annotated dataset, particularly for certain act classes. This made feature selection difficult and potentially limited the performance of more complex models.
  • Focus on Therapist: This work exclusively focuses on therapist dialogue acts. Future work should also analyze client statements to gain a more comprehensive understanding of the therapeutic conversation.
  • Exploration of Advanced Techniques: The authors suggest further exploration of semisupervised learning, active learning for more efficient labeling, and more principled ensemble methods.
  • Annotation Checking: The potential for using these models to identify potentially misannotated sentences is highlighted as a future avenue.
  • Broader Applicability: Investigating how to generalize these findings to even more diverse therapeutic approaches and larger datasets is a key future direction.

Summary updated

Abstract

While conversation in therapy sessions can vary widely in both topic and style, an understanding of the underlying techniques used by therapists can provide valuable insights into how therapists best help clients of different types. Dialogue act classification aims to identify the conversational "action" each speaker takes at each utterance, such as sympathizing, problem-solving or assumption checking. We propose to apply dialogue act classification to therapy transcripts, using a therapy-specific labeling scheme, in order to gain a high-level understanding of the flow of conversation in therapy sessions. We present a novel annotation scheme that spans multiple psychotherapeutic approaches, apply it to a large and diverse corpus of psychotherapy transcripts, and present and discuss classification results obtained using both SVM and neural network-based models. The results indicate that identifying the structure and flow of therapeutic actions is an obtainable goal, opening up the opportunity in the future to provide therapeutic recommendations tailored to specific client situations.
BibTeX
@misc{lee2019identifying,
  author = {Fei-Tzin Lee and Derrick Hull and Jacob Levine and Bonnie K. Ray and Kathy McKeown},
  title = {{Identifying therapist conversational actions across diverse psychotherapeutic approaches}},
  year = {2019},
  doi = {10.18653/v1/w19-3002},
}
Figure