TL;DR
This paper presents the University of Maryland's system for the CLPsych 2015 shared task, which aimed to classify Twitter users based on self-reported depression and PTSD. They achieved strong results by combining traditional TF-IDF features with novel supervised topic models (sLDA and sAnchor), demonstrating the power of capturing latent linguistic structures for mental health classification.
What Problem Does This Paper Address?
Identifying individuals with mental health disorders like depression and PTSD is a significant societal challenge. Traditional methods are often costly, intrusive, and limited by stigma or lack of self-awareness. This paper tackles the problem of using online language, specifically Twitter posts, as a low-cost, unobtrusive method for early screening and monitoring of these conditions. The CLPsych 2015 shared task provided a benchmark for evaluating systems on this task, and the authors aimed to contribute a robust and effective system to this effort.
Methods
The University of Maryland's system is built upon a supervised classification framework, primarily using Support Vector Machines (SVMs). They explored two main types of features:
- Lexical Features (TF-IDF): This is a standard approach where the importance of words in a document is weighted by their frequency within that document and inversely by their frequency across all documents. They experimented with both "short" and "big" vocabularies.
- Supervised Topic Models (sLDA and sAnchor): This is the core innovation. Instead of just using word frequencies, they employed topic models that learn latent themes within the text and also incorporate the known labels (depression, PTSD, control).
- sLDA (Supervised Latent Dirichlet Allocation): They used sLDA to learn topic distributions for each document. Crucially, they incorporated "informed priors" for sLDA, pre-training a 50-topic model on a corpus of stream-of-consciousness essays to guide the learning process on the Twitter data.
- sAnchor: They also explored a supervised variant of the "anchor" algorithm, which is designed for faster topic modeling and can identify an "anchor" word for each topic.
- Feature Configurations: They combined these topic model posteriors with TF-IDF features in various ways, including using only TF-IDF, only topic models, or a combination of both.
They also experimented with different SVM kernels (linear and RBF) and vocabulary sizes. For data preparation, they aggregated tweets by week to create documents, aiming to balance document size and temporal relevance.
Datasets
- CLPsych 2015 Shared Task Dataset: This dataset consists of Twitter users labeled with one of three categories: depression, PTSD, or control.
- Stream-of-consciousness essays (Pennebaker and King, 1999): Used to create "informed priors" for the sLDA model. This dataset consists of essays written by college students.
Key Insights and Ideas
- Supervised Topic Models Enhance Performance: Combining traditional TF-IDF features with supervised topic model posteriors significantly improved classification performance over TF-IDF alone. This highlights the value of capturing latent thematic structures that go beyond simple word counts.
- Linear SVMs Excel: For this task, linear SVMs generally outperformed RBF kernel SVMs, suggesting that the decision boundaries are relatively linear in the feature space.
- Vocabulary Size Matters (for Linear SVMs): Using a larger vocabulary ("big vocabulary") improved the performance of linear SVMs, but not RBF SVMs. This indicates that a richer set of words can be beneficial when the model can effectively utilize it.
- Depression vs. PTSD is More Challenging: The results indicated that distinguishing between depression and PTSD was more difficult than distinguishing either from controls. This suggests a greater overlap in linguistic signals between these two conditions.
- Topic Interpretability Offers Clinical Value: While the primary goal was classification, the authors found that the learned topics from sLDA were often interpretable and could potentially offer insights to clinicians, providing face validity and suggesting new avenues for research. For example, topics related to entertainment and social isolation were linked to depression.
Key Results
The best-performing system, which used a big vocabulary, combined all features (TF-IDF and topic models), and employed a linear SVM, achieved an average Area Under the Curve (AUC) of approximately 0.84 across the three tasks. Specifically, the best AUCs were:
- Depression vs. Control (DvC): 0.841
- Depression vs. PTSD (DvP): 0.832
- PTSD vs. Control (PvC): 0.879
The paper also shows ROC curves for all submitted systems, illustrating the competitive performance of their top models.
Limitations and Future Work
- Weekly Aggregation Limitations: Aggregating tweets by week, while improving performance, means that a user's mental health status might not be accurately reflected for every week in their data. This could be mitigated by more precise temporal labeling of diagnoses.
- Further Error Analysis: The authors plan to conduct a detailed error analysis to understand where their models are making mistakes.
- Exploring Other Aggregations: They suggest exploring other time-based aggregations, such as grouping tweets by time of day, to see if performance can be further improved.
- More Sophisticated Topic Models: The paper hints at the potential for even more sophisticated topic models to enhance predictive accuracy and clinical insight.
- Informed Priors for sAnchor: They did not explore informed priors for the sAnchor algorithm, which could be a direction for future work.
Summary updated