Automatic Identification of Ruptures in Transcribed Psychotherapy Sessions

Adam Tsakalidis, Dana Atzil-Slonim, Asaf Polakovski, Natalie Shapira, Rivka Tuval-Mashiach, Maria LiakataNAACL 2021

TL;DR

This paper introduces the first NLP approach to automatically detect "ruptures" in psychotherapy sessions from transcribed dialogue. By training on client and therapist self-reports, their model significantly outperforms a baseline and can identify ruptures missed by therapists, offering a cost-effective way to improve therapeutic practice.

What Problem Does This Paper Address?

The therapeutic alliance, the relationship between a client and therapist, is a critical predictor of successful psychotherapy outcomes. When this alliance experiences "ruptures" – moments of tension, misunderstanding, or conflict – it can lead to poor outcomes or premature termination of therapy. While ruptures can be beneficial if recognized and repaired, they often go unnoticed. Traditional methods for identifying ruptures rely on self-reports, which are typically collected at a low temporal resolution (e.g., once per session) and have limitations like participant self-insight and response biases. Previous attempts to use within-session coding are labor-intensive and expensive. This paper addresses the need for an automated, fine-grained method to detect these crucial alliance ruptures directly from the therapy session's text.

Methods

The core of this work is an NLP-based classification task to identify alliance ruptures. They frame this as a binary classification problem: did a rupture occur in a given session? They train two separate models: one to predict client-reported ruptures (CR) and another for therapist-reported ruptures (TR).

The input to their models is the transcribed dialogue from therapy sessions. They represent each session using dense word vectors derived from a skip-gram model trained on Hebrew tweets. Specifically, they create three types of representations for each session: client text only, therapist text only, and a concatenation of both.

For classification, they employ Logistic Regression. To ensure robust evaluation and generalizability, they use a leave-one-client-out cross-validation strategy. Their primary evaluation metric is the macro-averaged F-score, which balances precision and recall across both rupture and non-rupture classes. They compare their model's performance against a simple majority baseline.

Datasets

The study utilizes a dataset of 873 transcribed psychotherapy sessions from 68 clients, collected between August 2014 and August 2016. These sessions were part of a university-based outpatient clinic's regular practice. Clients were selected based on treatment duration (at least 15 sessions) and availability of full data, including audio recordings and session-by-session questionnaires.

Crucially, the dataset includes self-reported alliance rupture labels from both clients and therapists. Ruptures were identified using a single question from the Post-Session Questionnaire (PSQ): "Did you experience any tension, misunderstanding, conflict or disagreement in the relationship with your client/therapist?" rated on a 5-point Likert scale. A rating higher than 1 was considered a rupture.

The final dataset used for experiments consists of 849 transcribed sessions. The distribution of labels is imbalanced for the client-reported rupture task (23% rupture vs. 77% no rupture), while the therapist-reported rupture task is more balanced (48% rupture vs. 52% no rupture).

Key Insights and Ideas

  • NLP as a "Rupture Detector": The paper demonstrates that NLP can effectively identify alliance ruptures from text, offering a scalable alternative to manual coding.
  • Client-Missed Ruptures: A significant finding is that their model can identify client-reported ruptures that therapists missed. This highlights the potential for automated tools to alert therapists to subtle or unrecognized ruptures, facilitating timely intervention.
  • Therapist Training Advantage: The dataset for therapist-reported ruptures is more balanced, suggesting therapists are generally better at recognizing ruptures, likely due to their training. This also implies that automated systems could be particularly valuable for identifying ruptures that are less obvious to human clinicians.
  • Withdrawal vs. Confrontational Ruptures: The qualitative analysis suggests that therapists are better at identifying "confrontational" ruptures (client expresses anger/dissatisfaction) than "withdrawal" ruptures (client disengages or avoids). This aligns with existing literature and further emphasizes the utility of NLP for detecting subtler forms of rupture.

Key Results

  • The proposed NLP baseline significantly outperforms a strong majority baseline across both client-reported rupture (CR) and therapist-reported rupture (TR) tasks.
  • For the TR task, the model achieved a macro-average F-score of 46.70%, a substantial improvement over the majority baseline's 37.50%.
  • For the CR task, the model achieved an F-score of 45.60%, also outperforming the majority baseline's 37.50%.
  • The model successfully identified 40% of client-reported ruptures that were missed by therapists.
  • When considering sessions where both client and therapist agreed on the absence of rupture, the models jointly achieved 60% correct classification.
  • For sessions with clear-cut labels (agreement between client/therapist and model prediction), the macro-average F1-score reached 70.9% with 83.6% accuracy.

Limitations and Future Work

  • Linguistic and Cultural Homogeneity: The study acknowledges that the clients and therapists come from similar linguistic and cultural backgrounds (Hebrew speakers). Future work should validate these findings on data from therapy sessions across different backgrounds.
  • Contextual Language Models: The authors plan to incorporate contextual language models (like BERT) to improve performance, suggesting that current word vector averaging might not fully capture the nuances of dialogue.
  • Sequential and Temporally Sensitive Models: The current models treat each session independently. Future research will focus on developing models that can capture the sequential and temporal dynamics within therapy sessions.
  • Bias in Methodology and Evaluation: The authors note that assessing mental health through social media and smart devices raises concerns about bias in methodology and evaluation, a broader challenge that likely extends to this work.

Summary updated

BibTeX
@inproceedings{tsakalidis2021automatic,
  author = {Adam Tsakalidis and Dana Atzil-Slonim and Asaf Polakovski and Natalie Shapira and Rivka Tuval-Mashiach and Maria Liakata},
  title = {{Automatic Identification of Ruptures in Transcribed Psychotherapy Sessions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure