Detecting Cognitive Distortions from Patient-Therapist Interactions

Sagarika Shreevastava, Peter FoltzNAACL 2021

TL;DR

This paper explores using Natural Language Processing (NLP) to automatically detect cognitive distortions, negative thinking patterns common in mental health issues like depression and anxiety. They found that Sentence-BERT embeddings combined with an SVM classifier achieved the best results for detecting the presence of distortions, highlighting the potential for automated mental health support tools.

What Problem Does This Paper Address?

Cognitive Behavioral Therapy (CBT) is a widely used approach for treating mental health conditions, and a core component of CBT is identifying and restructuring negative thinking patterns known as cognitive distortions. With the rise in mental health issues and the increasing use of virtual therapy, there's a growing need for accessible and affordable mental health support. This paper addresses the challenge of automatically detecting these cognitive distortions from patient-therapist interactions using NLP. The goal is to build tools that can assist in identifying these patterns, potentially making mental health support more scalable and efficient.

Methods

The researchers investigated various NLP features and machine learning algorithms for detecting cognitive distortions. They categorized features into semantic (SIF, Sentence-BERT) and syntactic (LIWC, POS tag embeddings). For classification, they experimented with logistic regression, support vector machines (SVM), decision trees, K-Nearest Neighbors (k-NN), and Multi-Layer Perceptrons (MLP). A key aspect was comparing the effectiveness of different feature types and algorithms, especially on a limited dataset. They also explored the linguistic characteristics associated with different types of distortions.

Datasets

The primary dataset used is the "Therapist Q&A" dataset, a crowd-sourced collection of anonymized patient-therapist interactions from Kaggle. This dataset consists of patient entries describing their circumstances and symptoms, followed by therapist responses. For annotation, the researchers focused on the patient's input. They annotated 3000 samples, classifying them into "No distortion" or one of ten specific cognitive distortion types. The inter-annotator agreement for specific distortion types was noted as a challenge, with a relaxed agreement rate of around 40%, but increased to 61% when distinguishing between distorted and non-distorted thinking.

Key Insights and Ideas

  • Semantic features shine: Sentence-BERT embeddings, which capture semantic meaning and word order, proved highly effective, suggesting that understanding what is said is crucial.
  • Word order less critical for detection: The study found that word order had little impact on the performance of semantic features for detecting the presence of distortions, implying that the core meaning is more important than sentence structure in this binary classification task.
  • Syntactic features offer linguistic clues: While not as effective for binary detection as semantic features, syntactic features (like LIWC and POS tags) provided valuable insights into the linguistic characteristics of different distortions, aiding in understanding why certain patterns are indicative of distortions.
  • Distortion type is harder to classify: While detecting the presence of distortions was feasible, accurately classifying the specific type of distortion proved significantly more challenging, with low accuracy scores. This suggests that the nuances between different distortions are subtle and difficult for current models to discern.
  • Inter-annotator agreement is a hurdle: The subjective nature of identifying cognitive distortions led to challenges in achieving high inter-annotator agreement, which in turn sets an upper bound on model performance.

Key Results

  • Best binary detection: The SVM classifier using pre-trained Sentence-BERT embeddings achieved the highest F1-score of 0.79 for detecting the presence of cognitive distortions.
  • Hybrid models: Combining semantic and syntactic features yielded similar results to using semantic features alone, suggesting that the semantic information was dominant.
  • Type classification performance: For detecting the type of cognitive distortion, none of the implemented ML techniques achieved an F1-score higher than 0.30.
  • Linguistic patterns: Analysis of POS tags revealed distinct patterns for certain distortions (e.g., "labeling" associated with adjectives and interjections, "mind-reading" with pronouns). LIWC features also showed differential behavior across distortion types, with "no distortion" entries exhibiting the most deviation from the norm.

Limitations and Future Work

The authors acknowledge the limitations of a small, annotated dataset and the challenges in achieving high inter-annotator agreement, which impacts the upper bound of classification performance. They also note that the "Therapist Q&A" dataset, while providing context, is still a patient's self-reported version of events, which might itself be distorted.

Future work includes implementing a multi-class classification framework to improve the accuracy of detecting specific distortion types. They also plan to make the annotated dataset publicly available to encourage further research. Additionally, they aim to train models using IOB (inside-outside-between) tagging to pinpoint specific parts of sentences that give rise to cognitive distortions, moving towards more granular error detection.

Summary updated

BibTeX
@inproceedings{shreevastava2021detecting,
  author = {Sagarika Shreevastava and Peter Foltz},
  title = {{Detecting Cognitive Distortions from Patient-Therapist Interactions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure