Biased Bots: An Empirical Demonstration of How AI Bias Could Compromise Mental Healthcare

Adela C. Timmons, Kexin Feng, Jacqueline B. Duong, Kayla E. Carta, Sierra Walters, Grace Jumonville, Alyssa S. Carrasco, Gabrielle F. Freitag, Daniela Romero, Gayla Margolin, Shrikanth Narayanan, Jonathan S. Comer, Matthew W. Ahle, Theodora ChaspariPsyArXiv 2024 · DOI

TL;DR

This paper provides a sobering empirical look at how AI models designed for mental health—specifically stress detection—can inherit and amplify societal biases. By testing a machine learning model on a diverse dataset of wearable and smartphone data, researchers found significant performance disparities: the model was less accurate for women and systematically overestimated stress levels in Hispanic/Latina participants, highlighting a critical risk of "pathologizing" normal behavior in marginalized groups.

What Problem Does This Paper Address?

The "mental health gap" is a global crisis, and many look to AI-driven Just-in-Time Adaptive Interventions (JITAIs) as a solution. These systems use smartphones and wearables to detect shifts in a user's state (like rising stress) and deliver support in the moment.

However, AI is only as good as its training data. If the data reflects a biased society or lacks diversity, the resulting algorithms may fail the very people who need help most. While AI bias is well-documented in facial recognition and hiring, its impact on mental healthcare—where a "false positive" could lead to misdiagnosis or unnecessary intervention—remains under-explored. This paper seeks to quantify that bias in the context of multimodal stress sensing.

Methods

The researchers developed a Twin Neural Network (TNN), a metric learning approach particularly suited for "few-shot learning." This is crucial in mental health apps where you might only have a few labeled data points (e.g., 14 hours of self-reports) per user.

  • The Architecture: The TNN consists of two identical sub-networks that process pairs of samples to produce similarity scores. Each sub-network used 89 input units, two hidden layers with LeakyReLU activation, and a Sigmoid output layer.
  • Training/Validation: They used a "leave-one-couple-out" cross-validation strategy. To handle missing data, they used a simple imputation method (carrying over the previous hour's measurement).
  • Statistical Analysis: Beyond standard ML metrics (F1, precision, recall), they used mixed-effects ANOVAs and paired-sample t-tests to account for the nested structure of the data (individuals within couples) and to rigorously test for performance differences across demographic subgroups.

Datasets

The study utilized a newly collected dataset of 212 English-speaking young adults (106 heterosexual couples) aged 18-25.

  • Data Streams: 24 hours of at-home data including smartphone audio (speech content and vocal quality), electrodermal activity (Q-sensors), and heart rate (Actiwaves).
  • Ground Truth: Hourly self-reported stress levels collected via smartphone surveys.
  • Features: A total of 89 physiological, vocal, and speech features were extracted.
  • Diversity: The sample included Asian, Black/African American, Hispanic/Latine, Multiracial, and White non-Hispanic/Latine participants.

Key Insights and Ideas

  • The "Clean Data" Privilege: One fascinating hypothesis for why the model performed better on men (higher F1 and precision) is physiological. Men generally have greater muscle mass, which can make wearable sensor data more resilient to "motion artifacts" (noise caused by movement). This suggests that even the physics of our sensors might favor certain body types.
  • Pathologizing through Overestimation: The model didn't just fail to detect stress; it "hallucinated" it. It significantly overestimated stress frequency in women, particularly Hispanic/Latina women. In a clinical setting, this could lead to "over-pathologizing" healthy individuals, reinforcing negative stereotypes.
  • The Intersectionality Paradox: While the model performed worst for White non-Hispanic/Latina women, it actually performed better for Black/African American women than other female subgroups. The authors suggest this might be because Black women in the sample faced unique, distinct stressors that created more "identifiable" biopsychological patterns for the algorithm to latch onto.
  • Representation Matters, but isn't a Silver Bullet: White non-Hispanic/Latina women had the largest representation in the sample but the lowest model performance. This suggests that simply having "more data" isn't enough if that data contains high variability that the model cannot distinguish from noise.

Key Results

  • Overall Performance: The model achieved a macro F1 score of 0.63, which is considered moderate—better than chance, but with significant room for improvement.
  • Gender Gap: Men saw significantly higher F1 scores ($M=0.54$) and precision ($M=0.55$) compared to women ($F1: M=0.46$; precision: $M=0.46$).
  • Stress Frequency Bias: For women, the algorithm-generated stress ratio ($M=0.52$) was significantly higher than the true self-reported ratio ($M=0.40$). This effect was most pronounced in Hispanic/Latina women (predicted $M=0.64$ vs. true $M=0.44$).

Limitations and Future Work

The authors are transparent about several hurdles:

  • Sample Constraints: The $N=212$ sample, while diverse, was still relatively small and unevenly distributed, limiting the power to detect subtle intersectional effects.
  • Binary/Heteronormative Focus: The study only included heterosexual couples and treated gender as a binary, which misses the experiences of LGBTQ+ individuals.
  • Linguistic Bias: The speech measures were English-only, likely biasing the model against non-native speakers.
  • Future Directions: The researchers call for "fair-aware AI" methodologies, such as sample reweighting and feature regularization, to be integrated into the development cycle of mental health tools. They also emphasize the need for interdisciplinary teams that combine ML expertise with social justice perspectives.

Summary updated

Abstract

Background: The proliferation of artificial intelligence (AI) applications for mental health has advanced in recent years and shows promise to increase the reach, scope, and impact of mental healthcare. However, biases in algorithms designed to assess and treat mental health problems pose risk for equitable mental health. This cross-sectional study investigates the existence of bias in algorithms for detecting stress from mobile devices and its implications for mental health equity.Methods: A diverse sample of young adults (N = 212) carried smartphones, wore physiological sensors, and completed hourly surveys assessing their subjective stress for 24 hours. We then developed a Twin Neural Network machine learning (ML) model to detect hourly stress from smartphone and wearable data and evaluated model performance across gender and ethnic/racial status. Findings: The model performed moderately well overall yet showed significant variation in performance ranging from poor to good across gender and ethnic/racial groups. In particular, the model evidenced lower performance in women compared to men and overestimated the frequency of stress episodes for Hispanic/Latina women.Interpretation: Findings highlight the presence of bias in AI applications for mental health and underscore the need for cautious interpretation of ML outcomes in historically underrepresented groups. Discussion focuses on the implications of AI bias for mental health and the importance of developing methods that combine AI and social justice perspectives to ensure the implementation of equitable mental healthcare.Funding: This project is based on work supported by NIMH Grant No. R42MH123368 (Timmons, Comer, Ahle, Co-PIs), NSF GRFP Grant No. 1930019 (Timmons, PI), NSF Grant No. BCS-1627272 (Margolin, PI), SC CTSI (NIH/NCATS) Grant No. UL1TR000130 (Margolin, PI), NIH-NICHD Grant No. R21HD072170-A1 (Margolin, PI), NSF GRFP Grant No. DGE-0937362 (Timmons, PI), an APA Dissertation Award (Timmons, PI), NSF GRFP Grant No. DGE-0937362 (Han, PI), and NSF Grant No. 2046118 (Chaspari, PI).Keywords: Artificial intelligence; bias; equitable mental health; multimodal stress sensing
BibTeX
@misc{timmons2024biased,
  author = {Adela C. Timmons and Kexin Feng and Jacqueline B. Duong and Kayla E. Carta and Sierra Walters and Grace Jumonville and Alyssa S. Carrasco and Gabrielle F. Freitag and Daniela Romero and Gayla Margolin and Shrikanth Narayanan and Jonathan S. Comer and Matthew W. Ahle and Theodora Chaspari},
  title = {{Biased Bots: An Empirical Demonstration of How AI Bias Could Compromise Mental Healthcare}},
  year = {2024},
  doi = {10.31234/osf.io/7t98e},
  howpublished = {PsyArXiv},
}
Figure