Automatic Detection of Psychological Distress Indicators and Severity Assessment from Online Forum Posts

Shirin Saleem, Rohit Prasad, Shiv Vitaladevuni, Maciej Pacula, Michael Crystal, Brian Marx, Denise Sloan, Jennifer Vasterling, Theodore SperoffCOLING 2012

TL;DR

This paper introduces a novel system for automatically detecting psychological distress indicators and assessing their severity from online forum posts. By leveraging multi-label classification and probabilistic logic with domain-specific rules, the system aims to identify individuals who may need clinical attention, addressing the challenge of under-diagnosis and the stigma associated with mental health issues.

What Problem Does This Paper Address?

Psychological disorders are a significant societal burden, yet they are often under-diagnosed and under-treated due to stigma and the time-consuming nature of traditional diagnostic methods. Online forums offer a unique, albeit noisy and informal, environment where individuals may openly discuss their struggles. This paper tackles the challenge of automatically analyzing these forum posts to:

  1. Detect specific psychological distress indicators that align with established diagnostic criteria (DSM-IV).
  2. Assess the severity of distress to prioritize individuals for clinical intervention (triage). The informal nature of forum text, including misspellings, grammatical errors, and vague language, presents significant hurdles for natural language processing (NLP). Furthermore, incorporating expert psychological knowledge is crucial for accurate assessment.

Methods

The proposed system employs a multi-stage text classification approach:

  • Relevance Classification: An initial SVM classifier filters out messages that are not relevant to psychological distress.
  • Feature Extraction: A rich set of features is extracted from the forum posts, going beyond simple bag-of-words. These include:
  • Lexical features: Unigrams, normalized pronoun count, punctuation count, average sentence length.
  • Sentiment features: Sentiment-bearing words from established lexicons (LIWC, General Inquirer).
  • Structural features: Indicators for lead author posts and first responder posts.
  • Thread-level features: Similarity measures between a message and other messages in the thread, or the first message.
  • Domain Phrase Features: Novel features derived from expert rationales provided for distress label annotations. These are n-grams identified from rationales that strongly correlate with specific distress labels.
  • Multi-label Distress Classification: Support Vector Machines (SVMs) are used in a "one-versus-all" binary classification setup to predict the presence or absence of each of the 136 fine-grained distress labels.
  • Probabilistic Logic-based Triage: Markov Logic Networks (MLNs) are employed to integrate text features, predicted distress labels, and domain-specific rules (derived from DSM-IV and PC-PTSD guidelines) to estimate triage codes. MLNs offer robustness to noise and allow for learning the weights of domain rules from data.

Datasets

The study utilizes a corpus of forum threads downloaded from an online forum for veterans discussing post-combat psychological issues, specifically PTSD and mTBI.

  • Corpus Size: 512 threads, comprising approximately 5000 relevant and irrelevant messages.
  • Annotation:
    • Distress Labels: 136 fine-grained labels derived from DSM-IV constructs, annotated by psychologists.
  • Rationales: For a subset of messages, contextual rationales supporting the distress labels were provided by expert psychologists.
  • Triage Codes: Authors were assigned triage codes (TR1, TR2, TR3) indicating treatment acuity, with TR1 (imminent danger) being rare in this forum. The focus was on distinguishing TR2 and TR3.

Key Insights and Ideas

  • Rationales as Features: Incorporating domain-specific phrases extracted from expert rationales significantly improves multi-label classification accuracy (14.6% improvement over plain text features). This highlights the value of expert justification in improving automated systems.
  • MLNs for Triage: Markov Logic Networks prove effective for triage assessment by combining text information, predicted distress labels, and expert-defined rules, demonstrating statistically significant gains over SVMs. This approach handles noisy label detections and potential rule contradictions.
  • Informal Text Challenges: The study acknowledges the inherent difficulties in processing informal text, such as misspellings, grammatical errors, and vague language, and proposes features to mitigate these.
  • Inter-annotator Agreement: While good agreement was found for triage codes (Kappa=0.71), agreement for distress labels varied, with some labels showing moderate agreement (Kappa=0.59-0.68) and others being more challenging due to the need for inference and world knowledge.

Key Results

  • Multi-label Distress Classification: The system achieved a mean AUC improvement of 14.6% by incorporating a comprehensive set of features, particularly the domain phrase features derived from rationales, over a baseline of unigram TF-IDF features.
  • Triage Classification: MLNs incorporating text, distress labels, and DSM-IV/PC-PTSD rules achieved the highest AUC of 0.4515 (bounded at 33% false acceptance rate), outperforming SVMs and MLNs using only text or text plus distress labels.
  • Class Imbalance: The dataset exhibits significant class imbalance, with some distress labels having very few training examples, impacting performance for those labels.

Limitations and Future Work

  • Limitations:
    • The study acknowledges the class imbalance in the dataset, which affects the performance for less frequent distress labels.
  • Inter-annotator agreement for some distress labels was only moderate, indicating inherent ambiguity or complexity in their annotation.
  • The focus was on distinguishing TR2 and TR3 triage codes due to the rarity of TR1 in the forum.
  • Future Work:
    • Investigate methods to exploit label dependencies in classification.
  • Incorporate contextual features that consider previous messages within a thread.
  • Validate the system on text data from subjects formally diagnosed with PTSD and compare outcomes with a control group.
  • Extend the system to detect TR1 triage codes if appropriate training data becomes available.

Summary updated

BibTeX
@inproceedings{saleem2012automatic,
  author = {Shirin Saleem and Rohit Prasad and Shiv Vitaladevuni and Maciej Pacula and Michael Crystal and Brian Marx and Denise Sloan and Jennifer Vasterling and Theodore Speroff},
  title = {{Automatic Detection of Psychological Distress Indicators and Severity Assessment from Online Forum Posts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure