Mental Illness Detection at the World Well-Being Project for the CLPsych 2015 Shared Task

Daniel Preotiuc-Pietro, Maarten Sap, H. Andrew Schwartz, Lyle H. Ungar

TL;DR

This paper presents a system for automatically detecting mental illnesses (PTSD and depression) from Twitter data. By combining user metadata with various text-based features derived from word clusters, their approach achieved a second-place ranking in the CLPsych 2015 shared task, demonstrating strong performance, especially at low false positive rates.

What Problem Does This Paper Address?

Mental illnesses are a significant global health concern, yet many individuals do not seek help due to stigma or imperfect screening methods. Social media, particularly platforms like Twitter, offers a unique window into users' psychological states through their online behavior. The CLPsych 2015 shared task aimed to automatically identify users who self-disclosed diagnoses of post-traumatic stress disorder (PTSD) and depression on Twitter. This is a challenging problem because it requires distinguishing between control groups and individuals with specific mental health conditions using naturally occurring, often noisy, text data.

Methods

The researchers employed a multi-faceted approach to tackle this classification problem. Their system combines two main types of features:

  • Metadata Features: These include user profile information such as the number of followers, friends, tweets, and account age, which can offer indirect insights into a user's online presence and potentially their social engagement.
  • Textual Features: This is where the core innovation lies. Instead of using raw word counts (unigrams), which can lead to sparse data and overfitting, they explored various word clustering techniques to represent text. These clusters group semantically or syntactically related words, effectively creating "topics" that can provide a more robust and lower-dimensional representation of user posts. They experimented with:
  • Brown Clusters: A hierarchical clustering method based on word co-occurrence.
  • NPMI Word Clusters: Using normalized pointwise mutual information to measure word similarity.
  • Word2Vec (W2V) and GloVe: Neural network-based word embedding techniques that capture contextual information.
  • LDA (Latent Dirichlet Allocation) and LDA ER: Topic modeling approaches that represent documents as mixtures of topics.

These diverse textual features were then fed into standard linear classifiers, specifically Logistic Regression and Linear Support Vector Machines. To further improve performance and reduce classifier variance, they implemented an ensemble method. This involved training a meta-classifier (logistic regression) on the predicted scores from individual classifiers, effectively learning to weigh the contributions of each feature set.

Datasets

The primary dataset used was provided by the CLPsych 2015 shared task organizers. This dataset consisted of 1,145 Twitter users labeled as either Controls, Depressed, or suffering from PTSD. The users had self-disclosed their mental illness diagnoses on Twitter. The researchers pre-processed this data by removing users with fewer than 500 words in their posts to mitigate sparsity issues.

Key Insights and Ideas

  • Word Clustering is Key: Simply using raw unigrams, while performing well, was not as robust as using features derived from word clusters. This highlights the importance of capturing semantic relationships between words for this task.
  • Ensembling for Robustness: Combining predictions from classifiers trained on different feature sets (especially those capturing diverse linguistic aspects) led to consistent performance improvements, suggesting that different feature types offer complementary information.
  • Task Agnosticism: The system was designed to be largely independent of specific medical keywords or data, making it potentially applicable to identifying other types of mental health conditions or even other behavioral patterns.
  • Performance at Low False Positives: The system demonstrated exceptional performance when the acceptable rate of false positives was very low, a crucial aspect for practical applications in clinical settings where misclassification can have serious consequences.

Key Results

The system achieved a second-place ranking in the CLPsych 2015 shared task.

  • On average precision, their best models scored above 0.80 for all three binary classification tasks (Controls vs. Depressed, Controls vs. PTSD, and Depressed vs. PTSD).
  • Specifically, they reached a high of 0.869 average precision for predicting PTSD from controls.
  • At a false positive rate of 0.1, their system achieved a true positive rate of 0.8 for separating Controls from PTSD users and 0.75 for separating Controls from depressed users. This performance at low false positive rates ranked them first in that specific evaluation metric.

Limitations and Future Work

The authors acknowledge that distinguishing between PTSD and depressed users was more challenging, particularly at small false positive rates.

Future work suggested includes:

  • Domain Adaptation: Utilizing other datasets with similar labels for different illnesses to improve generalization and prediction quality.
  • Learning to Rank: Exploring "learning to rank" algorithms as an alternative to traditional classifiers, which might be more suitable for ranking users based on their likelihood of having a particular condition.

Summary updated

Abstract

This article is a system description and report on the submission of the World Well-Being Project from the University of Pennsylvania in the ‘CLPsych 2015’ shared task. The goal of the shared task was to automatically determine Twitter users who self-reported having one of two mental illnesses: post traumatic stress disorder (PTSD) and depression. Our system employs user metadata and textual features derived from Twitter posts. To reduce the feature space and avoid data sparsity, we consider several word clustering approaches. We explore the use of linear classifiers based on different feature sets as well as a combination use a linear ensemble. This method is agnostic of illness specific features, such as lists of medicines, thus making it readily applicable in other scenarios. Our approach ranked second in all tasks on average precision and showed best results at .1 false positive rates.
BibTeX
@inproceedings{preotiucpietro2015mental,
  author = {Daniel Preotiuc-Pietro and Maarten Sap and H. Andrew Schwartz and Lyle H. Ungar},
  title = {{Mental Illness Detection at the World Well-Being Project for the CLPsych 2015 Shared Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/w15-1205},
}
Figure