Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA & China

Joseph Cameron, Jiaee Cheong, Micol Spitale, Hatice Gunes

Abstract

Social agents and robots are increasingly being used in wellbeing settings. However, a key challenge is that these agents and robots typically rely on machine learning (ML) algorithms to detect and analyse an individual's mental wellbeing. The problem of bias and fairness in ML algorithms is becoming an increasingly greater source of concern. In concurrence, existing literature has also indicated that mental health conditions can manifest differently across genders and cultures. We hypothesise that the representation of features (acoustic, textual, and visual) and their inter-modal relations would vary among subjects from different cultures and genders, thus impacting the performance and fairness of various ML models. We present the very first evaluation of multimodal gender fairness in depression manifestation by undertaking a study on two different datasets from the USA and China. We undertake thorough statistical and ML experimentation and repeat the experiments for several different algorithms to ensure that the results are not algorithm-dependent. Our findings indicate that though there are differences between both datasets, it is not conclusive whether this is due to the difference in depression manifestation as hypothesised or other external factors such as differences in data collection methodology. Our findings further motivate a call for a more consistent and culturally aware data collection process in order to address the problem of ML bias in depression detection and to promote the development of fairer agents and robots for wellbeing.

Introduction

Depression manifestation is not universal; symptoms often vary significantly based on an individual's gender and cultural background. Despite this, most machine learning (ML) models for depression detection are developed using Western datasets, which may introduce "Northern American and European" biases. As social robots and AI wellbeing coaches are deployed globally, ensuring algorithmic fairness across demographic groups is critical.

This study addresses three primary questions:

  1. Do depression-related features (acoustic, textual, and visual) differ across genders and countries?
  2. How do different ML models and data modalities compare in performance across cultures?
  3. Are there significant gaps in model fairness across these demographic groups?

Methodology

The researchers employed a two-pronged approach. First, Statistical Analysis using one-way and two-way MANOVA (Multivariate Analysis of Variance) was conducted to determine if specific features—such as facial action units (FAUs), speech pitch, or head pose—differed significantly between depressed and healthy individuals, and between genders.

Second, Machine Learning Experimentation involved training seven classifiers (including SVM, Logistic Regression, and Multi-Layer Perceptrons) on various modality combinations: Acoustic (A), Textual (T), and Visual (V). Fairness was evaluated using Equal Accuracy (EA) (the accuracy gap between genders) and Disparate Impact (DI) (the ratio of positive predictions between genders).

Datasets

The study compared two major multimodal datasets:

  1. CMDC (Chinese Multimodal Depression Corpus): 78 participants. Data was collected via semi-structured clinical interviews, with labels based on the PHQ-9.
  2. E-DAIC (Extended Distress Analysis Interview Corpus): A US-based dataset with 275 participants. Data was collected via casual recorded conversations, with labels based on the PHQ-8.

Both datasets provided acoustic features via openSMILE and visual features (FAUs, head pose, gaze) via OpenFace.

Results and Analysis

Cross-Cultural Feature Manifestation

Statistical analysis revealed that depression "looks" different across borders. In the Chinese CMDC dataset, depressed individuals showed distinct facial action units and head orientations, while their gaze patterns remained similar to healthy controls. Conversely, in the US E-DAIC dataset, gaze was a significant differentiator for depression, while other facial features were less conclusive.

Performance and the "Elicitation" Effect

Models performed significantly better on the Chinese CMDC dataset, reaching F1-scores up to 0.98, compared to the US E-DAIC dataset, which peaked around 0.80. This discrepancy suggests an "elicitation effect": the structured clinical interviews in CMDC may elicit more consistent behavioral markers than the casual conversations in E-DAIC.

Performance and fairness metrics for the Chinese CMDC dataset.

As shown in the CMDC results, SVM Linear and Logistic Regression were highly effective, particularly when combining Textual and Visual features. In contrast, the US dataset required more robust models like the Multi-Layer Perceptron (MLP) to maintain performance across modalities.

Performance and fairness metrics for the US E-DAIC dataset.

Fairness and Multimodality

Fairness metrics highlighted significant gaps, particularly in the US dataset where the Naive Bayes model showed a Disparate Impact (DI) as high as 1.76. However, the study found that multimodality stabilizes fairness. While unimodal models (audio-only or text-only) showed high variance in fairness, combining modalities (A+T+V) generally resulted in a narrower range of DI scores, suggesting that diverse data sources can help mitigate gender bias.

Conclusion

While differences in depression manifestation across cultures were observed, it remains unclear if these are purely cultural or artifacts of data collection methodologies (clinical vs. casual). The findings underscore the need for a unified, culturally aware clinical protocol for data collection. Such consistency is vital for developing fair and effective AI agents for global mental health support.

Summary updated

Abstract

Social agents and robots are increasingly being used in wellbeing settings. However, a key challenge is that these agents and robots typically rely on machine learning (ML) algorithms to detect and analyse an individual's mental wellbeing. The problem of bias and fairness in ML algorithms is becoming an increasingly greater source of concern. In concurrence, existing literature has also indicated that mental health conditions can manifest differently across genders and cultures. We hypothesise that the representation of features (acoustic, textual, and visual) and their inter-modal relations would vary among subjects from different cultures and genders, thus impacting the performance and fairness of various ML models. We present the very first evaluation of multimodal gender fairness in depression manifestation by undertaking a study on two different datasets from the USA and China. We undertake thorough statistical and ML experimentation and repeat the experiments for several different algorithms to ensure that the results are not algorithm-dependent. Our findings indicate that though there are differences between both datasets, it is not conclusive whether this is due to the difference in depression manifestation as hypothesised or other external factors such as differences in data collection methodology. Our findings further motivate a call for a more consistent and culturally aware data collection process in order to address the problem of ML bias in depression detection and to promote the development of fairer agents and robots for wellbeing.
BibTeX
@inproceedings{cameron2024multimodal,
  author = {Joseph Cameron and Jiaee Cheong and Micol Spitale and Hatice Gunes},
  title = {{Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA \& China}},
  booktitle = {ACII (Workshops and Demos)},
  year = {2024},
  doi = {10.1109/aciiw63320.2024.00054},
}
Figure