Identifying Distorted Thinking in Patient-Therapist Text Message Exchanges by Leveraging Dynamic Multi-Turn Context

Kevin Lybarger, Justin Tauscher, Xiruo Ding, Dror Ben-zeev, Trevor CohenNAACL 2022

TL;DR

This paper tackles the challenge of automatically identifying "cognitive distortions" – common thinking errors – in text message conversations between patients and therapists. By leveraging the conversational history (context), their BERT-based models significantly improve detection accuracy, bringing us closer to AI-powered tools for mental health support and therapist training.

What Problem Does This Paper Address?

Cognitive Behavioral Therapy (CBT) is a cornerstone treatment for various mental health conditions. A key aspect of CBT involves identifying and reframing "cognitive distortions," which are systematic errors in thinking that warp a person's perception of reality. While effective, identifying these distortions can be complex for clinicians, especially with the increasing use of remote therapy options like text messaging. This paper addresses the gap in automatically detecting these distortions within the dynamic, multi-turn nature of patient-therapist text message exchanges. Previous work often treated each message in isolation, missing crucial contextual cues that a human therapist would naturally use.

Methods

The authors propose using BERT-based architectures to identify six types of cognitive distortions in patient text messages. Their core innovation lies in incorporating conversational context. They explore two main BERT architectures:

  • BERT-only: This model treats the entire conversation (target message plus preceding context messages) as a single input sequence, with messages separated by a [SEP] token.
  • BERT+LSTM: Here, each message is first encoded by BERT individually, and then these encoded messages are fed into a Long Short-Term Memory (LSTM) recurrent neural network to capture sequential dependencies.

Crucially, they investigate dynamic methods for representing conversation context during training:

  • Fixed Context: A predefined number of preceding messages are always used.
  • Random Length Context: The number of context messages varies randomly during training, allowing the model to learn from contexts of different lengths.
  • Random Mask Context: Similar to random length, but some context messages are randomly masked (removed) during training.

These dynamic approaches aim to make the models more robust and better mimic how humans process information over time.

Datasets

The study utilizes a corpus of text message exchanges collected from a randomized controlled trial augmenting routine care for individuals with serious mental illness. This corpus consists of messages from 39 patients and 9 therapists, totaling 7,436 patient and 6,959 therapist messages. The patients had diagnoses including schizophrenia, schizoaffective disorder, major depressive disorder, or bipolar disorder. The dataset was manually annotated by a doctoral-level licensed mental health counselor and a master's-level psychologist for six specific cognitive distortion types: Catastrophizing (C), Jumping to conclusions (J), Mental filtering (M), Should statements (S), Overgeneralization (O), and Unspecified (U). An "Any distortion" (A) label was also assigned if at least one distortion was present.

Key Insights and Ideas

  • Context is King: The most significant finding is that incorporating conversational context dramatically improves the identification of cognitive distortions, outperforming models that treat messages in isolation.
  • Dynamic Context Matters: Dynamic context approaches (random length and random mask) offer a modest but statistically significant improvement over fixed context, suggesting that models benefit from learning to handle varying context lengths and potential missing information.
  • Clustering of Distortions: The analysis reveals that cognitive distortions tend to "cluster" in time, meaning they are more likely to appear in consecutive messages. This explains why context is so effective – it helps disambiguate messages that might seem innocuous in isolation.
  • BERT-only Shines: The BERT-only architecture, which encodes the entire conversation as a single sequence, achieved the best overall performance, particularly with the random length context.

Key Results

  • The BERT-only model with random length context achieved the highest performance for "Any distortion" (A) at 0.73 F1, a significant improvement over context-agnostic models (0.68 F1).
  • Context-aware models approached inter-rater agreement for a majority of distortion types.
  • While overall performance improved, detecting specific, less frequent distortion types (like catastrophizing or jumping to conclusions) still showed lower F1 scores (< 0.5).
  • The dynamic context methods (random length and random mask) provided a modest but statistically significant improvement over the fixed context for more frequent distortions.

Limitations and Future Work

  • Dataset Size and Diversity: The study is limited by the number of participating patients and therapists. The corpus was split such that patients appear in both train and test sets, which might limit generalizability. Future work should involve larger, more diverse datasets.
  • Annotation Challenges: Manually annotating distortions is inherently challenging, with soft boundaries between types. The authors are working on adding more fine-grained distortion categories and improving annotation guidelines.
  • Exploring Other Contexts: The paper focused on preceding conversational turns. Future work could explore other forms of context, such as message metadata (time of day, response times) or patient demographics, and investigate learned patient embeddings.
  • Beyond BERT: While BERT-based models performed well, exploring other transformer architectures or domain-specific pre-trained models could yield further gains.

Summary updated

BibTeX
@inproceedings{lybarger2022identifying,
  author = {Kevin Lybarger and Justin Tauscher and Xiruo Ding and Dror Ben-zeev and Trevor Cohen},
  title = {{Identifying Distorted Thinking in Patient-Therapist Text Message Exchanges by Leveraging Dynamic Multi-Turn Context}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}

Loading related content...

Figure