Understanding who uses Reddit: Profiling individuals with a self-reported bipolar disorder diagnosis

Glorianna Jagfeld, Fiona Lobban, Paul Rayson, Steven JonesNAACL 2021

TL;DR

This paper uses NLP to automatically profile nearly 20,000 Reddit users who self-report a bipolar disorder (BD) diagnosis, revealing demographic and comorbidity characteristics that differ from general populations. This work provides crucial context for BD research on Reddit and offers a model for ethically profiling online communities.

What Problem Does This Paper Address?

Research on mental health conditions using online data, particularly Reddit, is growing rapidly. However, a significant gap exists in understanding the characteristics of the users contributing this data. Without this information, it's difficult to assess the generalizability of findings from studies on platforms like Reddit to broader clinical or general populations. This paper aims to bridge this gap by automatically inferring clinical, demographic, and identity characteristics of Reddit users who self-report a bipolar disorder (BD) diagnosis, and comparing these to existing statistics.

Methods

The researchers employed a combination of pattern-matching and existing NLP methods to extract user characteristics. For user identification, they used pattern matching on posts to find self-reported BD diagnoses, distinguishing between professional diagnoses and self-diagnoses. To infer characteristics, they adapted and evaluated several NLP techniques:

  • Mental Health Comorbidities: Pattern matching was used to identify mentions of other DSM-5 and ICD-11 diagnoses within user posts.
  • Age: A hybrid approach was used, combining self-reported age from user posts (extracted via regular expressions) with a neural network model that predicts age group based on language use.
  • Country of Residence: A Dirichlet process mixture model was employed, analyzing word distributions, posting frequency per subreddit, and posting times to infer country.
  • Binary Gender: A hybrid approach was used, prioritizing username analysis, then self-reported gender, and finally language use to predict gender.

Datasets

The primary dataset consists of approximately 21.4 million public Reddit posts from nearly 20,000 user accounts that self-reported a bipolar disorder diagnosis. These posts were collected from January 5th to March 19th. For evaluation, the authors manually annotated subsets of users for age, country, and gender. They also referenced existing datasets and epidemiological studies for comparison, including:

  • The SMHD dataset (Coppersmith et al., 2018) for self-reported mental health comorbidities.
  • US survey data on Reddit users (Barthel et al., 2016) for age and gender comparisons.
  • Global Burden of Disease Study (2017) for BD prevalence rates.

Key Insights and Ideas

  • Reddit users with BD are not representative: The demographic and comorbidity profiles of these Reddit users significantly differ from general Reddit users and epidemiological studies, highlighting the need for careful interpretation of Reddit-based research.
  • The power of hybrid NLP: Combining multiple NLP methods (e.g., self-report + language use for age, username + self-report + language use for gender) significantly improves both accuracy and coverage for inferring user characteristics.
  • Ethical considerations are paramount: The paper strongly emphasizes the ethical challenges of online research, particularly concerning consent, anonymity, and the potential for misuse of inferred user characteristics.
  • BD on Reddit is complex: The data reveals a high prevalence of other mental health comorbidities among users who self-report BD, suggesting a complex interplay of conditions and potentially misdiagnoses.

Key Results

  • User Identification: Approximately 19,685 Reddit users with self-reported BD diagnoses were identified, contributing over 21 million posts.
  • Demographics:
    • Age: 80% of users were between 18-49 years old at their first post, with a mean age of 27.7 years.
  • Gender: 52% of users were identified as feminine, contrasting sharply with general Reddit username trends.
  • Country: Over 80% of users were estimated to live in the US, with other English-speaking countries following.
  • Comorbidities: 50.7% of users reported at least one additional mental health diagnosis. Major Depressive Disorder (MDD) was the most common (30.2%), followed by Anxiety Disorder (15.8%) and ADHD (12.9%).
  • NLP Method Performance: Hybrid methods achieved high accuracy (e.g., 99% for age, 97% for gender) and coverage for inferring user characteristics.

Limitations and Future Work

  • User Identification: The study acknowledges that the identified users likely represent only a fraction of all Reddit users with BD, due to the reliance on explicit self-reporting and pattern matching.
  • Self-Reported Data: The accuracy of self-reported diagnoses and characteristics is inherently limited, and the study cannot confirm diagnoses with clinical interviews.
  • NLP Limitations: The gender inference methods are limited to binary gender, excluding transgender and non-binary individuals, which is a significant limitation given the stigmatized nature of these identities. The methods for comorbidities do not distinguish between current, past, or misdiagnosed conditions.
  • Country Inference: The country inference method relies on recent posts and may not accurately reflect a user's current location if they have relocated.
  • Future Work: The authors suggest that Reddit can serve as a valuable platform for understanding under-researched comorbidities and for recruiting participants to clinical studies. They also highlight the need for NLP tools that can better identify a wider range of gender identities. The work also serves as a model for ethically profiling other online populations.

Summary updated

BibTeX
@inproceedings{jagfeld2021understanding,
  author = {Glorianna Jagfeld and Fiona Lobban and Paul Rayson and Steven Jones},
  title = {{Understanding who uses Reddit: Profiling individuals with a self-reported bipolar disorder diagnosis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure