Linking Language-based Distortion Detection to Mental Health Outcomes

Vasudha Varadarajan, Allison Lahnala, Sujeeth Vankudari, Akshay Raghavan, Scott Feltman, Syeda Mahwish, Camilo Ruggero, Roman Kotov, H. Andrew SchwartzNAACL 2025

TL;DR

This paper bridges the gap between automatically detecting cognitive distortions in text and their correlation with validated mental health outcomes. They found that higher rates of detected distortions are linked to greater mental health symptom severity, and specific distortion types like "should statements" are associated with conditions like depression. This work suggests that NLP-based distortion detection can be a valuable tool for monitoring and understanding mental health.

What Problem Does This Paper Address?

Cognitive distortions are systematic thinking errors that contribute to and maintain mental health conditions like depression and anxiety. Recognizing and reframing these distortions is a cornerstone of Cognitive Behavioral Therapy (CBT). While Natural Language Processing (NLP) has made strides in automatically detecting these distortions, a crucial missing piece has been validating whether these language-detected distortions actually correlate with established, real-world mental health outcomes. This paper aims to empirically bridge this gap, demonstrating that NLP-detected distortions are not just linguistic curiosities but meaningful indicators of mental well-being.

Methods

The researchers employed a two-pronged approach:

  1. Distortion Detection: They trained transformer-based language models (specifically, RoBERTa-base and DistilRoBERTa-base) to detect cognitive distortions in text. This involved two tasks:
  2. Binary Classification: Distinguishing between text containing any distortion versus text without.
  3. Multi-class Classification: Identifying specific types of distortions (e.g., "All-or-Nothing Thinking," "Fortune Telling"). They explored both "span-only" (focusing on the exact distorted phrase) and "full message" (considering the entire context) approaches.
  4. Correlation Analysis: Once distortions were detected in new datasets, they analyzed the co-occurrence of distortion rates with mental health outcomes. This was done at two levels:
  5. Between-user correlations: Examining if individuals with higher overall distortion rates also reported more severe mental health symptoms.
  6. Within-user correlations (limited): Exploring how changes in distortion rates over time might relate to changes in mental health states.

Datasets

The study utilized several datasets:

  • Training Data for Distortion Detection:
    • Patient Queries (PQ) dataset: Contains patient queries to therapists, annotated with 1-2 dominant cognitive distortions from 10 common types.
  • Thinking Traps (TT) dataset: Contains examples of situations leading to distorted thoughts, annotated with 13 types of cognitive distortions. (Note: Three distortion types were dropped from TT to align with PQ).
  • Mental Health Outcomes Data:
    • iHiTOP dataset: Transcribed clinical interviews from psychiatric outpatients, aligned with the HiTOP taxonomy. This dataset provides expert-reported scores across seven "spectra" (e.g., internalizing, thought disorder).
  • DS4UD dataset: Comprises daily diary entries from U.S. service industry workers, collected via Ecological Momentary Assessments (EMAs). This dataset includes self-reported daily metrics like mood, stress, and anxiety.

Key Insights and Ideas

  • Distortions as Real-World Markers: The most significant insight is that automatically detected cognitive distortions are not just linguistic artifacts but are demonstrably linked to actual mental health conditions. This validates the practical utility of NLP in this domain.
  • Severity Matters: Higher rates of detected distortions consistently correlate with greater severity of mental health symptoms. This suggests a dose-response relationship, where more distorted thinking implies more significant distress.
  • Specific Distortions, Specific Outcomes: The paper identifies that certain distortion types have stronger associations with particular mental health indicators. This hints at the potential for fine-grained analysis, where specific thinking patterns might serve as unique markers for certain conditions.
  • "Should Statements" and "Fortune Telling" as Key Indicators: These specific distortions showed notable correlations with depressed mood and emotional drain, highlighting them as particularly relevant for these conditions.

Key Results

  • Positive Correlations: Between-user correlation analyses showed that overall distortion rates positively correlated with mental health outcome scores across both the iHiTOP and DS4UD datasets. For instance, in iHiTOP, distortion rates correlated significantly with all seven spectra. In DS4UD, they correlated with depression, emotional drain, and nervous stress.
  • Specific Distortion Correlations:
    • "Should Statements" showed positive correlations with depressed mood and emotional drain (EMA) and depression (WAVE) in DS4UD.
  • "Fortune Telling" also showed positive correlations with depression (WAVE) and emotional drain (EMA).
  • Model Performance: The paper reports cross-validation metrics (F1 and AUC scores) for distortion detection and classification, indicating reasonable performance, with the TT + PQ (span) model achieving an F1 score of 0.833 for detection.

Limitations and Future Work

  • Model Performance Variability: The authors acknowledge that classification models showed variable performance across different distortion types, with some (like "mind reading" and "personalization") exhibiting particularly weak correlations.
  • Dataset Limitations: The analysis was limited to two datasets, which may have sampling biases. The study was also restricted to English language data.
  • Within-Person Analysis Constraints: The within-person analysis was limited by a small number of repeated measures, restricting the ability to observe detailed within-user fluctuations.
  • Future Directions:
    • Improved Detection: Further development is needed to better detect less common distortion types.
  • Longitudinal Studies: More longitudinal data is required to robustly study within-user fluctuations in mental health and thinking patterns.
  • Contextual Nuances: The reliance on automated methods may miss contextual nuances that human clinicians observe, suggesting these tools should be used as assistance rather than replacements.
  • Broader Frameworks: Exploring more complex frameworks for capturing nuanced distorted thinking is suggested.

Summary updated

BibTeX
@inproceedings{varadarajan2025linking,
  author = {Vasudha Varadarajan and Allison Lahnala and Sujeeth Vankudari and Akshay Raghavan and Scott Feltman and Syeda Mahwish and Camilo Ruggero and Roman Kotov and H. Andrew Schwartz},
  title = {{Linking Language-based Distortion Detection to Mental Health Outcomes}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure