Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA & China

Joseph Cameron, Jiaee Cheong, Micol Spitale, Hatice Gunes

TL;DR

This paper investigates gender fairness in AI-driven depression prediction models by comparing performance and bias across multimodal datasets from the USA and China. The study reveals that while depression manifestation differs across cultures and genders, it's not definitively clear if this is due to cultural differences or data collection methodologies, highlighting the need for more standardized and culturally aware data collection for fairer AI.

What Problem Does This Paper Address?

As AI-powered agents and robots become more integrated into mental wellbeing support, accurately and fairly detecting mental health conditions like depression is crucial. A significant challenge is that depression can manifest differently across genders and cultures, potentially leading to biases in machine learning (ML) models trained on existing datasets. Western-developed diagnostic tools, for instance, might be biased towards specific demographic groups. This paper tackles the critical issue of ML bias in depression detection, specifically by examining how gender and cultural differences impact model performance and fairness, a problem that has been largely under-explored across different countries.

Methods

The researchers employed a comprehensive approach involving statistical analysis and ML experimentation. They utilized a range of ML models, including SVM (Linear and RBF), Logistic Regression, Naive Bayes, KNN, Decision Tree, and MLP, to ensure their findings were not algorithm-dependent. For statistical analysis, they used one-way and two-way MANOVA tests to compare features across depressed vs. non-depressed groups and across genders. For ML evaluation, they used stratified 5-fold cross-validation and assessed models based on overall accuracy, F1 Score, and AUROC. Crucially, they also evaluated fairness using Equal Accuracy (EA) and Disparate Impact (DI) metrics, which measure accuracy gaps and outcome ratios between majority and minority groups, respectively.

Datasets

The study utilized two existing multimodal depression datasets:

  • Chinese Multimodal Depression Corpus (CMDC): This dataset contains textual, acoustic, and visual data from 78 Chinese participants. Ground truth labels were obtained using the Patient Health Questionnaire-9 (PHQ-9).
  • American Extended Distress Analysis Interview Corpus (E-DAIC): This US-based dataset includes textual, acoustic, and visual data from 275 participants. Ground truth labels were derived from the Patient Health Questionnaire-8 (PHQ-8).

The paper notes that while both datasets contain similar modalities, the data elicitation methods differed significantly (semi-structured clinical interviews for CMDC vs. more casual conversations for E-DAIC), which is identified as a key limitation.

Key Insights and Ideas

  • Cultural Nuances in Depression Manifestation: The study found statistically significant differences in how depression manifests across genders and cultures, particularly in visual and acoustic features. However, the strength of these differences varied between the US and Chinese datasets.
  • Multimodal Advantage for Fairness: On the CMDC dataset, multimodal approaches generally led to narrower Disparate Impact (DI) scores across genders compared to unimodal setups, suggesting that combining different data sources can improve fairness.
  • Model Consistency Across Datasets: Certain models, like SVM Linear and Logistic Regression, consistently performed well across both datasets and modalities, while others, such as Naive Bayes, consistently underperformed.
  • Data Collection Methodology Matters: The significant performance differences observed between the CMDC and E-DAIC datasets, despite using similar ML models and modalities, suggest that the data collection methodology (e.g., semi-structured interviews vs. casual conversations) might be a more significant factor than cultural differences alone in explaining these disparities.
  • The "Fairness" Challenge: While models showed varying degrees of fairness, the paper highlights that achieving consistent fairness across different cultural contexts remains a significant challenge.

Key Results

  • Performance: SVM Linear and Logistic Regression models generally achieved the highest performance (accuracy, F1, AUROC) across various feature combinations on the CMDC dataset. On the E-DAIC dataset, the MLP model often showed the best performance. Naive Bayes consistently performed the worst on both datasets.
  • Fairness (CMDC): For the CMDC dataset, models like SVM Linear and Logistic Regression demonstrated good fairness, achieving EA scores of 0.00 and DI scores of 1.00 in several configurations. Naive Bayes consistently showed the least fair outcomes.
  • Fairness (E-DAIC): Across the E-DAIC dataset, fairness results were more varied. While some models showed good fairness, Naive Bayes again produced the least fair outcomes across most features and modalities.
  • Feature Differences: Statistical analysis revealed significant differences in visual (facial action units, head pose) and acoustic (pitch, amplitude, spectral) features between depressed and non-depressed participants in CMDC. For E-DAIC, significant differences were noted in gaze features, but not acoustic features.

Limitations and Future Work

  • Data Collection Methodology: The authors acknowledge that the significant differences in data collection methods between the CMDC and E-DAIC datasets make it difficult to definitively attribute performance disparities to cultural differences in depression manifestation.
  • Sample Size: The limited number of participants in both datasets, particularly for the underprivileged group, is noted as a limitation.
  • Future Work: The paper calls for the creation of new, standardized multimodal depression datasets collected across multiple countries using a single, clinically approved protocol. They also suggest exploring other evaluation methods beyond stratified k-fold cross-validation and investigating additional fairness metrics. Furthermore, they emphasize the need for developing solutions to mitigate bias and encourage culturally aware data collection practices to promote fairer ML-equipped agents and robots for wellbeing.

Summary updated

Abstract

Social agents and robots are increasingly being used in wellbeing settings. However, a key challenge is that these agents and robots typically rely on machine learning (ML) algorithms to detect and analyse an individual's mental wellbeing. The problem of bias and fairness in ML algorithms is becoming an increasingly greater source of concern. In concurrence, existing literature has also indicated that mental health conditions can manifest differently across genders and cultures. We hypothesise that the representation of features (acoustic, textual, and visual) and their inter-modal relations would vary among subjects from different cultures and genders, thus impacting the performance and fairness of various ML models. We present the very first evaluation of multimodal gender fairness in depression manifestation by undertaking a study on two different datasets from the USA and China. We undertake thorough statistical and ML experimentation and repeat the experiments for several different algorithms to ensure that the results are not algorithm-dependent. Our findings indicate that though there are differences between both datasets, it is not conclusive whether this is due to the difference in depression manifestation as hypothesised or other external factors such as differences in data collection methodology. Our findings further motivate a call for a more consistent and culturally aware data collection process in order to address the problem of ML bias in depression detection and to promote the development of fairer agents and robots for wellbeing.
BibTeX
@inproceedings{cameron2024multimodal,
  author = {Joseph Cameron and Jiaee Cheong and Micol Spitale and Hatice Gunes},
  title = {{Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA \& China}},
  booktitle = {ACII (Workshops and Demos)},
  year = {2024},
  doi = {10.1109/aciiw63320.2024.00054},
}

Loading related content...

Figure