TL;DR
This paper investigates why machine learning models trained to detect depression from social media text exhibit biases across demographic groups. By performing a qualitative analysis of topics discussed by different groups, the authors reveal that non-clinical, temporal topics and dataset imbalances significantly contribute to model errors, rather than inherent differences in how depression is expressed.
What Problem Does This Paper Address?
Machine learning models are increasingly used to identify mental health conditions, like depression, from social media data. However, these models often perform worse for minority demographic groups (e.g., people of color, women) compared to majority groups. While dataset underrepresentation is a known factor, this paper explores a less-understood cause: whether depression is expressed differently across demographics, leading to model generalization issues. The authors aim to qualitatively analyze the content of social media posts from various demographic groups to understand these differences and their impact on model performance.
Methods
The core of this research is a qualitative analysis using a Partially Labeled Latent Dirichlet Allocation (PLDA) model. This is an extension of the standard LDA topic modeling technique. Unlike standard LDA, PLDA allows for the incorporation of pre-defined labels (in this case, depression/control status and demographic group) directly into the topic modeling process. This enables the identification of "label-specific topics" – topics that are particularly prevalent within a specific demographic group or within the depressed subgroup of a demographic.
The authors also employ two key metrics:
- Enrichment (E) metric: Measures how distinctive a topic is for a particular group.
- Non-normalized average difference (Δ): Compares the prevalence of a topic between depressed and control groups within a demographic. Positive Δ values indicate topics more aligned with depression.
- Topic error rate (Ê): Identifies topics that are disproportionately present in misclassified instances, highlighting areas where models struggle.
Datasets
The study utilizes two existing Twitter datasets for depression identification:
- CLPsych 2015 Shared Task: Contains publicly available tweets from individuals who self-reported depression.
- Multi-disorder multitask learning for mental health dataset: A larger dataset combining subsets from various sources, also using self-reported depression.
Both datasets were pre-processed to remove numeric values, usernames, retweets, and URLs. English stopwords, pronouns, and emojis were also filtered to create more coherent topics.
Key Insights and Ideas
- Non-clinical topics drive differences: The most significant content differences between depressed and control subgroups across demographics are often due to non-clinical phenomena like discussions about TV shows, awards, or popular culture trends.
- Temporal topics mislead models: Topics related to short-term events (e.g., "People's Choice Awards," "One Direction" fandom) are prevalent in model errors. These topics are not generalizable and can cause models to misclassify individuals based on temporary trends rather than genuine depression signals.
- Dataset balance is crucial: The underrepresentation of certain demographic groups (e.g., Male PoC) can lead to spurious topics being associated with depression for those groups, as the model overfits to limited data. Conversely, over-represented control topics (e.g., "beauty") can also skew results.
- Language use isn't the primary differentiator: While some demographic-specific language (like AAVE) appears, it's not the main driver of depression expression differences. Instead, the context and temporal nature of discussions are more influential.
- Demographic label noise: The presence of non-English language topics or minority accent topics in unexpected demographic groups (e.g., Arabic on Female-White) suggests potential noise in demographic inference methods, highlighting the need for self-reported labels.
Key Results
The qualitative analysis revealed that while there are content differences between depressed and control subgroups across demographics, these are largely driven by non-clinical and temporal topics. For instance, topics like "One Direction" and "5 Seconds of Summer" were found to be prevalent in depression subgroups for certain demographics but not others, and were also correlated with model errors. The study found that topics prevalent in model errors were often those not representative across demographics or influenced by temporal phenomena. The authors also noted that dataset size and balance play a significant role in these content differences and model performance.
Limitations and Future Work
The authors acknowledge limitations in their demographic labels, which were inferred rather than self-reported, and the limited size of some demographic groups in the datasets. They suggest that future work should focus on:
- Developing more inclusive datasets with a wider range of demographic axes (e.g., age, economic status).
- Utilizing self-reported demographic labels for more accurate analysis.
- Investigating methods to generalize temporal topics and mitigate their impact on model performance.
- Expanding this qualitative analysis methodology to other mental health disorders and datasets.
Summary updated