Identifying Depression on Reddit: The Effect of Training Data

Inna Pirina, Çağrı ÇöltekinEMNLP 2018

TL;DR

This paper explores how the choice of training data significantly impacts the accuracy of identifying depression from social media text. By experimenting with various Reddit subreddits, the authors demonstrate that using topically specific data can lead to misleadingly high performance, while more realistic, general-purpose data presents a greater challenge but offers more generalizable insights.

What Problem Does This Paper Address?

Clinical depression is a serious mental health condition that often goes undiagnosed and untreated. Identifying depression through language analysis, particularly from social media, holds promise as a supplementary diagnostic tool or for monitoring mental well-being. However, a key challenge is obtaining suitable data for training and evaluating these identification models. Previous research has often relied on data from specialized forums or social media posts explicitly discussing depression. This paper questions whether models trained on such "easy" data generalize well to identifying depression in everyday language, where individuals might not explicitly mention their condition. They aim to investigate the impact of data source selection on the reliability and generalizability of depression detection models.

Methods

The authors employ Support Vector Machines (SVMs) with a bag-of-n-grams approach (both character and word n-grams) as their primary classification method. This is a standard and effective technique for text classification. They extract features from the text and weight them using the BM25 algorithm. The SVM margin parameter (C) and the order of n-grams are optimized through random search with 5-fold cross-validation. For experiments with imbalanced datasets, class weights are used during training. The implementation relies on Python and the scikit-learn library.

Datasets

The paper utilizes several datasets, primarily collected from Reddit:

  • DSF (Depression Support Forums): Data from Ramirez-Esparza et al. (2008), consisting of 400 forum posts by individuals seeking support for depression.
  • DND (Non Depression Forums): Gorbunova (2017)'s dataset, used as a negative class, with 400 posts from a breast cancer forum.
  • DS (Depression Support subreddit): Posts from a Reddit community dedicated to depression.
  • BC (Breast Cancer subreddit): Posts from a Reddit community for breast cancer discussion, used as a control.
  • FF (Family and Friends subreddits): Posts from subreddits related to family and friendship advice, chosen for their potential topical similarity to depression discussions.
  • DO (Depression - Other forums): A newly constructed dataset. This involves identifying users who explicitly mention being diagnosed with depression on the DS subreddit and then collecting their other posts from various subreddits within a month. This aims to capture depression in a more general context.
  • ND (No Depression - Other forums): A control dataset constructed similarly to DO, but using posts from authors who did not post in the depression subreddit during the investigation period.

All datasets generally consist of 400 training instances, with DO and ND also having an additional 400 for testing.

Key Insights and Ideas

  • Data Source is Crucial: The most significant finding is that the choice of training and testing data is paramount. Models trained on data explicitly discussing depression (e.g., DS-BC) achieve very high performance, but this performance can be misleading and does not necessarily translate to identifying depression in general language.
  • The "Easy" vs. "Hard" Data Divide: Datasets where the positive class (depression) and negative class are topically distinct and explicit (like DS-BC) lead to inflated F1 scores. Conversely, identifying depression in general-topic posts from diagnosed individuals (DO) against healthy individuals (ND) presents a much harder, but more realistic, challenge.
  • Reddit as a Valuable Resource: The paper highlights Reddit's potential for mental health research due to its anonymity and the availability of diverse communities.
  • Simplicity Can Be Effective: Linear SVMs with bag-of-n-grams features, despite their simplicity, perform competitively and are more interpretable than complex deep learning models.

Key Results

The paper reports F1 scores, with the highest scores achieved on highly specific datasets. For instance, the DS-BC model reached an F1 score of 98.62% in 5-fold cross-validation. However, when tested on the more challenging DO-ND setting (identifying depression in general language), the performance dropped significantly. The best performing model on the DO-ND test set was the forum data (DSF-NDF) with an F1 score of 64.05%, while the DO-ND specific setting yielded 67.49%. Combining all positive and negative datasets (excluding DO and ND) resulted in a model that performed comparably but worse than more specific models, though it showed slightly better performance on the general test set.

Limitations and Future Work

The authors acknowledge that their DO dataset might contain false positives due to not manually checking all sampled texts. They also note that the F1 scores obtained on Reddit, while higher than some previous Twitter-based studies, are lower than others, potentially due to the smaller number of training instances in their study. Future work could involve investigating these differences further and exploring the interpretability of the linear models to understand which linguistic features are most indicative of depression. The authors also suggest that further research is needed to understand the differences in performance across various social media platforms.

Summary updated

BibTeX
@inproceedings{pirina2018identifying,
  author = {Inna Pirina and Çağrı Çöltekin},
  title = {{Identifying Depression on Reddit: The Effect of Training Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure