On the Utility of Audiovisual Dialog Technologies and Signal Analytics for Real-time Remote Monitoring of Depression Biomarkers

Michael Neumann, Oliver Roessler, David Suendermann-Oeft, Vikram RamanarayananACL 2020

TL;DR

This paper explores using a cloud-based multimodal dialog system (NEMSI) to analyze speech and video for real-time remote depression monitoring. They found that specific audio and visual cues extracted during these conversations can significantly differentiate individuals with varying levels of depression, suggesting a promising avenue for scalable mental health assessment.

What Problem Does This Paper Address?

Diagnosing and monitoring mental health conditions like depression is crucial but faces significant hurdles. Access to specialists can be limited, awareness of conditions may be low, standardized diagnostic tools are not always readily available, and traditional in-person assessments involve substantial logistical and financial burdens. Furthermore, a shortage of medical specialists exacerbates these issues. The paper aims to bridge this gap by leveraging readily available technology (web browsers, mobile phones) to enable continuous, remote monitoring of depression biomarkers in everyday environments, moving beyond traditional lab settings.

Methods

The core of the methodology revolves around the NEMSI (NEurological and Mental health Screening Instrument) system. NEMSI is a cloud-based multimodal dialog system designed for automated screening interviews. Key aspects of their approach include:

  • Ubiquitous Access: NEMSI utilizes devices accessible to everyone (web browsers, mobile apps, phones), eliminating the need for specialized hardware.
  • Scalable Cloud Backend: The system's backend is deployed in a scalable cloud environment, allowing for cost-effective service to a large number of users.
  • Real-time Audiovisual Analytics: NEMSI is equipped with modules that extract a variety of speech and video features in real-time. These features are informed by existing literature on depression biomarkers.
  • Speech Feature Extraction: Metrics like speaking rate, pause duration, fundamental frequency (F0), volume, and jitter are extracted using the Praat software. Heuristics are applied to filter out obvious outliers.
  • Visual Feature Extraction: Facial landmark detection (using Dlib) identifies 68 key points, from which 14 specific facial metrics are calculated. These include measures of lip opening, mouth width, eyebrow displacement, eye opening, and eye blinks.
  • Statistical Analysis: A non-parametric Kruskal-Wallis test is used to determine statistically significant differences (p < 0.05) in these metrics between participant cohorts defined by their PHQ-8 depression scores.

Datasets

The study collected data from 307 interactions. After automatic cleaning (removing sessions with missing PHQ-8 answers, relevant speech metrics, or no detected faces), 208 sessions remained for analysis. The participants' depression severity was assessed using the Patient Health Questionnaire eight-item depression scale (PHQ-8), which served as the ground truth for their analysis. No new benchmark datasets were introduced; rather, existing data was collected and analyzed.

Key Insights and Ideas

  • Real-world utility of remote monitoring: The study demonstrates that audiovisual cues captured through a readily accessible dialog system can indeed serve as meaningful biomarkers for depression, even in less controlled environments.
  • Specific biomarkers emerge: Certain speech and visual metrics show significant differences across depression severity levels, suggesting that not all metrics are equally informative, and targeted feature selection is key.
  • Context matters for speech analysis: The type of speech sample (free speech vs. read speech vs. sustained vowels) influences which acoustic measures are most effective in differentiating depression severity.
  • Facial cues are sensitive to depression: Specific facial movements and static features, particularly related to the eyes and mouth, appear to be sensitive indicators of depression severity, especially in free speech contexts.

Key Results

The paper reports that:

  • For read speech, no significant differences were found for any of the analyzed metrics.
  • For free speech, percentage of pause time and volume were significant indicators for distinguishing depression groups.
  • For sustained vowels, significant differences were observed in volume, mean fundamental frequency, harmonics-to-noise ratio, and jitter.
  • Visual metrics showed significant differences for free and read speech. For males, eye-related metrics (eye_open, eyebrow_vpos, eye_blinks) were consistently significant. For all samples or females, the significance of visual metrics varied with cutpoints and speech type, with cutpoint 10 often including movement measures.

Limitations and Future Work

The authors acknowledge several limitations:

  • Self-reported PHQ-8 scores: These may be less accurate than clinician-assessed scores.
  • Generalizability: The study's findings need further validation across larger and more diverse populations, considering factors like age, native language, and device quality.
  • Input signal quality: Real-world conditions can introduce noise and variability that might affect metric reliability.

Future work proposed includes:

  • Collecting data from larger and more diverse populations.
  • Performing additional analyses such as regression analysis.
  • Extending the set of investigated metrics.
  • Investigating the efficacy of the system for other neurological or mental health conditions.

Summary updated

BibTeX
@inproceedings{neumann2020utility,
  author = {Michael Neumann and Oliver Roessler and David Suendermann-Oeft and Vikram Ramanarayanan},
  title = {{On the Utility of Audiovisual Dialog Technologies and Signal Analytics for Real-time Remote Monitoring of Depression Biomarkers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure