Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnoses

Keith Harrigian, Mark DredzeNAACL 2022

TL;DR

This paper investigates how long self-disclosed depression diagnoses remain valid indicators of mental health on social media. By analyzing a decade of Twitter data, they found that the predictive power of these diagnoses diminishes over time, suggesting that static labels are insufficient for longitudinal mental health analysis. This has critical implications for how we build and use datasets for computational mental health research.

What Problem Does This Paper Address?

Computational mental health research heavily relies on self-disclosed diagnoses from social media as ground truth for training models. However, mental health conditions are dynamic and can change due to treatment or other factors. The core problem this paper addresses is the assumption that a single, self-disclosed diagnosis label remains relevant and accurate over extended periods. This assumption can lead to models that are not robust and do not accurately reflect the longitudinal nature of mental health, potentially leading to flawed insights and interventions. The authors question the validity of using historical self-disclosures as a static representation of an individual's current mental state.

Methods

The researchers employed a mixed-methods approach. First, they updated and re-analyzed the 2015 CLPsych Shared Task dataset, which contains self-disclosed depression diagnoses and control groups from Twitter. They used a standard longitudinal domain transfer setup, dividing the data into three distinct three-year periods (2012-2015, 2015-2018, 2018-2021). They trained binary logistic regression classifiers on each period and evaluated their performance both within and across these time periods using Monte Carlo Cross Validation.

Second, to interpret model performance variations, they conducted a qualitative analysis. An annotator was presented with up to 30 tweets from individuals and asked to determine if they exhibited evidence of depression, considering DSM-5 criteria and prior knowledge. This analysis aimed to understand what the statistical classifiers were learning and to identify potential confounds in the data.

Datasets

The primary dataset used is the 2015 CLPsych Shared Task dataset. The authors created an "updated version" of this dataset, extending its temporal scope to cover data from 2012 to 2021. This updated dataset includes more recent activity from individuals who originally disclosed a depression diagnosis. The original dataset was constructed by identifying candidate self-disclosures using regular expressions and then having experts verify them. Control groups were sampled randomly to match the depression group's inferred age and gender.

Key Insights and Ideas

  • Temporal Relevance is Crucial: Self-disclosed diagnoses are not static. Their relevance as ground truth diminishes significantly over time, meaning models trained on older data may not generalize well to newer data.
  • "Then" vs. "Now" Matters: The study highlights that a diagnosis disclosed in the past might not reflect the current mental state, especially when considering treatment or remission.
  • Dataset Curation Needs Nuance: The findings underscore the need for more sophisticated dataset curation strategies beyond simple self-disclosure identification. This includes considering the timing of disclosures and potential confounding factors.
  • Personality and Comorbidities as Confounds: The analysis revealed that personality traits and comorbid psychiatric conditions are prevalent in self-disclosed depression data and can act as confounds, potentially inflating predictive performance or misdirecting model learning.
  • "Oversharing" as a Signal: Interestingly, individuals disclosing depression sometimes exhibited "oversharing" of personal topics, which, while potentially a sign of seeking connection, can complicate the task of identifying depression-specific signals.

Key Results

  • Within-Time-Period Performance Degradation: The average test-set area under the curve (AUC) for models trained and tested within the same time period was significantly higher for the original period (2012-2015) compared to subsequent periods. For example, within-period performance dropped from 0.71 in 2012-2015 to 0.66 in 2015-2018 and 0.68 in 2018-2021.
  • Between-Time-Period Generalization: While models trained on older data showed minimal performance loss when applied to newer data, models trained on newer data incurred a slight loss when applied to the original time period. However, these differences were minimal, suggesting that while temporal relevance shifts, the core signals might not entirely disappear.
  • Decreased Evidence of Depression: Qualitative analysis showed a significant decrease in the prevalence of linguistic evidence of depression over time. Approximately 76% of individuals in the original depression group showed clear evidence in 2012-2015, dropping to 45% in 2015-2018 and 39% in 2018-2021.
  • Annotator Bias: The inter-rater reliability analysis revealed cognitive biases among annotators, particularly with one annotator being more prone to use the "Uncertain" label and another showing higher sensitivity to negative emotions.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Sample Size for Annotation: The qualitative analysis was based on a limited number of tweets (up to 30 per individual), which might not capture the full picture of an individual's mental state.
  • Potential for Annotator Insufficiency: The annotation procedure might not have provided annotators with sufficient information or criteria to accurately indicate evidence of depression.
  • Dataset Specificity: The findings are based on a single dataset (CLPsych) and may not generalize to other social media platforms or cultural groups.
  • Confounding Factors: The study suggests that personality traits and comorbid conditions act as confounds, but further research is needed to fully disentangle these effects from genuine depression signals.

Future work suggested includes:

  • Broader Data Collection: Expanding analyses to multiple social media platforms, cultural groups, and time periods to draw more conclusive statements about the validity of self-disclosed diagnoses.
  • Improved Dataset Curation: Developing methods to annotate diagnosis dates and psychiatric comorbidities to better capture the temporal relevance of disclosures.
  • Propensity Score Matching: Recommending the use of propensity score matching for control groups to mitigate self-disclosure biases.
  • Bias Identification: Developing and applying algorithmic approaches to identify and filter sampling biases in datasets.
  • Investigating Remission: Further exploring the dynamics of depression remission and how it impacts self-disclosed labels.

Summary updated

BibTeX
@inproceedings{harrigian2022then,
  author = {Keith Harrigian and Mark Dredze},
  title = {{Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnoses}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure