Abstract
Depression has robust natural language correlates, and can increasingly be measured in language using predictive models. However, despite evidence that language use varies as a function of individual demographic features (e.g., age, gender), previous work has not systematically examined whether and how depression’s association with language varies by race. Here, we examined how race moderates the relationship between language features (i.e., first-person pronouns and negative emotions) and self-reported depression, in a matched sample of Black and White participants. Analyses revealed moderating effects of race: while depression severity predicts I-usage in White individuals, it does not in Black individuals. Negative emotion language use varies by race; as depression severity increased, White individuals used more belongingness and self-deprecation language, while Black individuals used more loneliness and worry language. Machine learning models trained to predict depression severity performed poorly when tested on Black individuals, even when they were trained exclusively using the language of Black individuals. In contrast, analogous models tested on White individuals performed relatively well. We ensured an equal number of Black and White individuals with similar age and gender distribution in our sample, thereby confirming that the performance disparity is not due to sampling bias. Our study reveals surprising race-based differences in the expression of psychological traits, like depression, in natural language, and highlights the need to better understand these effects, especially before language-based models for detecting psychological phenomena are integrated into clinical practice.
Introduction
Computational linguistics has long promised a future where social media data provides a scalable, low-cost way to screen for mental health disorders. Previous research established that depressed individuals tend to use more "I-talk" (self-focus) and more negative emotion words. However, most of this foundational research was conducted on samples that were either racially homogenous or didn't account for race at all. This study addresses the "generalizability crisis" in mental health NLP. If the way we express distress is culturally or racially situated, then models trained on majority populations will inevitably misdiagnose or overlook marginalized groups, exacerbating existing healthcare disparities.
Methods
The researchers employed a matched-pairs design to ensure that differences weren't due to age or gender. They recruited 868 English speakers in the U.S. (434 Black, 434 White) who shared their Facebook status updates and completed the PHQ-9 depression severity scale. The team used three primary analytical approaches: dictionary-based analysis (LIWC-2022) to measure word frequencies, Latent Dirichlet Allocation (LDA) for topic modeling, and machine learning models (Ridge regression and BERT embeddings) to predict depression scores in a cross-racial "2x2" design.
Results
The "I" in Depression is Race-Dependent
In White participants, higher depression scores were linearly associated with increased use of first-person singular pronouns. For Black participants, this correlation was significantly weaker. This suggests that the "self-focus" theory of depression may not manifest linguistically in the same way across cultures.
Predictive Accuracy Gap
The study revealed a stark disparity in model performance. Models trained and tested on White individuals achieved a Pearson correlation of $r = 0.392$. In contrast, models trained and tested on Black individuals only reached $r = 0.126$. Even high-capacity transformer models (BERT) showed the same disparity, with the strongest correlations always appearing in the White test sets, regardless of the training data.
Discussion
These findings highlight a critical blind spot in digital mental health: the role of race in how depression is expressed through language. The "gold standard" linguistic markers used to detect depression work well for White individuals but fail significantly for Black individuals. This suggests that the features typically used in these models may simply be less predictive of depression for Black users on Facebook compared to White users. This research highlights a major fairness gap in "digital phenotyping" that must be addressed before these tools enter clinical use. Future work must include more diverse populations and investigate the underlying causes of these differences, whether they stem from cultural manifestations of depression or differing social media performances.
Summary updated