Automated Speech Analysis for Risk Detection of Depression, Anxiety, Insomnia, and Fatigue: Algorithm Development and Validation Study

Rachid Riad, M. Denais, Marc de Gennes, A. Lesage, Vincent Oustric, Xuan Cao, Stéphane Mouchabac, Alexis BourlaJournal of Medical Internet Research 2024 · DOI

TL;DR

This paper presents a novel machine learning system that uses speech analysis to detect and estimate the severity of depression, anxiety, insomnia, and fatigue in the general population. The system demonstrates promising accuracy and, importantly, addresses crucial aspects like fairness across demographics and the ability to abstain when uncertain, paving the way for responsible clinical deployment.

What Problem Does This Paper Address?

Mental health assessment is challenging due to the heterogeneous and co-occurring nature of symptoms like depression, anxiety, insomnia, and fatigue. Traditional assessment methods, relying on questionnaires, are time-consuming and not always practical for widespread screening. While speech analysis shows promise as an objective, non-invasive biomarker, existing research often focuses on single symptoms and overlooks critical factors for real-world application, such as model fairness across different demographic groups and the system's ability to indicate uncertainty. This paper aims to bridge this gap by developing and validating a comprehensive speech-based system that addresses these multifaceted needs.

Methods

The researchers developed a novel speech and machine learning pipeline. This involved:

  • Speech Encoding: Utilizing pretrained deep learning models (specifically, the Whisper M model) to extract rich feature representations from speech. These models were pretrained on a large, open-source speech database.
  • Model Training: Fine-tuning these speech encoders for specific mental health tasks (depression, anxiety, insomnia, fatigue). They explored different pooling strategies (max pooling) and data balancing techniques (oversampling) to optimize performance.
  • Clinical Threshold Detection: Training classification models to identify individuals above predefined clinical thresholds for each condition.
  • Severity Estimation: Developing regression models to predict the actual severity scores of these conditions, moving beyond simple binary classification.
  • Fairness Assessment: Evaluating the model's performance across different demographic groups (sex, age, education level) using disparity ratios (DR) to ensure equitable service quality.
  • Selective Prediction: Implementing a mechanism for the model to abstain from making a prediction when it is highly uncertain, a crucial feature for safe clinical deployment.

Datasets

  • Callyope General-Population (Callyope-GP) Dataset: This is the primary dataset used in the study, comprising speech recordings and self-reported mental health assessments from 865 healthy adult participants. The dataset was split into training, validation, and testing sets.
  • Large non-annotated speech database: Used for pre-training the speech encoder models.
  • Clinically validated questionnaires: Used for collecting self-reported mental health data, including the Beck Depression Inventory (BDI), Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder 7-item scale (GAD-7), Athens Insomnia Scale (AIS), and Multidimensional Fatigue Inventory (MFI).

Key Insights and Ideas

  • Multisymptom Detection is Feasible: The study demonstrates that speech analysis can effectively capture signals related to multiple, often co-occurring, mental health conditions simultaneously, moving beyond single-symptom approaches.
  • Beyond Binary Classification: Predicting the severity of symptoms, not just whether a threshold is met, offers a more nuanced understanding and is achievable with speech data.
  • Fairness and Uncertainty are Crucial: The paper highlights that for any AI system to be clinically useful and ethical, it must perform equitably across demographics and be able to signal when it's not confident in its prediction. This is a significant step towards responsible AI in healthcare.
  • Co-occurrence of Insomnia and Fatigue: The study uniquely addresses the combined impact of fatigue and insomnia on speech, a common comorbidity that has been under-explored in speech-based mental health research.

Key Results

  • Detection Performance: The system achieved good detection performance for all symptoms, with F₁-scores ranging from 0.43 (MFI reduced motivation) to 0.88 (MFI total fatigue score) and AUCs for depression detection reaching up to 0.78.
  • Severity Estimation: Pearson correlations between predicted and actual scores were significant across all conditions, ranging from 0.31 to 0.49, indicating that the models could estimate symptom severity.
  • Fairness: The models showed good consistency for sex (average DR 0.86), moderate consistency for education level (average DR 0.47), and lower consistency for age groups (average DR 0.33).
  • Selective Prediction: The risk-coverage AUCs for selective prediction were generally low, indicating good abstention capabilities, particularly for depression (BDI) and fatigue.

Limitations and Future Work

  • Dataset Limitations: The study used a French monolingual, medium-sized dataset, which limits generalizability. The non-longitudinal design also misses insights into symptom evolution.
  • Self-Assessment Bias: Reliance on self-assessment scales introduces potential bias as it depends on participants' self-insight.
  • Fixed Train-Dev-Test Split: While useful for comparison, this approach can introduce bias and limit generalizability.
  • Future Work: The authors suggest future research should focus on larger, more diverse samples, longitudinal designs, inclusion of pathological data, and exploring various speech tasks and prompts to enhance user adherence and broaden the system's applicability. They also aim to further improve fairness and uncertainty mitigation methods.

Summary updated

Abstract

This study demonstrates the potential of speech-based systems for multifaceted mental health assessment in the general population, not only for detecting clinical thresholds but also for estimating their severity. Addressing fairness and incorporating uncertainty estimation with selective classification are key contributions that can enhance the clinical utility and responsible implementation of such systems.
BibTeX
@article{riad2024automated,
  author = {Rachid Riad and M. Denais and Marc de Gennes and A. Lesage and Vincent Oustric and Xuan Cao and Stéphane Mouchabac and Alexis Bourla},
  title = {{Automated Speech Analysis for Risk Detection of Depression, Anxiety, Insomnia, and Fatigue: Algorithm Development and Validation Study}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/58572},
}
Figure