Abstract
Introduction: Previous studies have established that depressive syndromes can be detected using machine learning methods, with multimodal data being essential. Multimodal data facilitates the extraction of characteristics such as gaze tracking, a reliable depression indicator. Our study employs high-quality video and other multimodal data from patients diagnosed with depression. Our study uses a multimodal data collection system (MDC) to understand the complex indicators of depression.
Objective: This paper outlines our protocol for deploying a multimodal data collection system within an In-Person Clinical Assessment environment. The system gathers high-definition videos, real-time vital signs, and voice recordings for future extraction of critical information such as eye gaze patterns. We aim to scale our model to provide portable depression risk analyses, facilitating timely intervention and encouraging patients to seek professional assistance.
Methods and Analysis: We have conducted sessions with 70 participants diagnosed with depression. Each participant undergoes DSM-5 interviews and engages with our multimodal data collection system. Participants respond to five on-screen scales while being recorded. To our knowledge, no other protocol has combined multimodal data collection and various stimuli in depression data collection.
Ethics and Dissemination: Ethical approval was provided by the National Health Commission of the PRC, Hefei Fourth People’s Hospital Ethics Committee (HSY-IRB-YJ-YYYX-JYF001). Results will be published in a peer-reviewed journal and presented at academic conferences.
Introduction
Diagnosing Major Depressive Disorder (MDD) remains a significant clinical challenge. Current diagnostic standards rely heavily on DSM-5 criteria and clinical interviews, which are inherently subjective and time-consuming. Patients may unintentionally or deliberately mask symptoms, and the scarcity of trained mental health professionals—particularly in developing regions—limits the accessibility of timely care.
While machine learning and computer vision have shown promise in identifying objective "instinctive" indicators of depression, such as eye-tracking patterns, most existing solutions require expensive, specialized hardware. This creates a barrier to widespread clinical adoption. To address this, researchers have developed a low-cost, multimodal data collection (MDC) protocol designed to identify objective behavioral markers using affordable commodity hardware.
The Multimodal Data Collection (MDC) System
The MDC system is designed to be non-wearable and cross-platform, ensuring a naturalistic environment for the patient. The setup utilizes three types of Hikvision webcams to capture different levels of detail: near-field cameras for facial micro-expressions, far-field cameras for general body behavior, and spatial location cameras to track movement within the room. Consistent illumination is maintained via a Xiaomi light bar, while a Berrymed patient monitoring system records real-time physiological data, specifically heart rate and SpO2 levels.
Experimental Protocol and Stimuli
The study involved 70 participants (aged 12 and older) with confirmed MDD diagnoses. The protocol is structured around exposing participants to specific emotional triggers to capture a synchronized record of their physiological and behavioral responses. These stimuli are categorized into four types:
- Images: 90 pre-selected images with positive, neutral, and negative emotional valences.
- Scales: Five digital psychiatric scales (SDS, PHQ-15, BAI-21, GAD-7, and SSI).
- Text: 90 words with varying emotional markers.
- Videos: A selection of comedy, tragedy, and neutral clips.
Initial pilot runs revealed that a 10-scale protocol was too fatiguing for patients. The researchers optimized the session by selecting five essential scales, reducing the total session time from 30 minutes to approximately 9 minutes. This refinement significantly improved the patient experience while maintaining data integrity.
Key Insights and Technical Feasibility
A core hypothesis of this research is that while patients can mask feelings during verbal interviews, their ocular trajectories and physiological responses to emotional stimuli are much harder to fake. The team successfully implemented the Multi-task Cascaded Convolutional Networks (MTCNN) algorithm to extract facial features and approximate gaze direction from standard near-field video feeds. This demonstrates that high-quality research data can be obtained without five-figure eye-tracking rigs, making the system highly scalable.
Furthermore, the protocol successfully synchronized video, audio, and vital sign streams into a single database. This provides a robust "ground truth" dataset for training future machine learning models to identify subtle, objective markers of depression.
Limitations and Future Work
As this research is in its early stages, current limitations include a relatively small sample size (70 participants) and the controlled nature of the clinical environment. Future work will focus on expanding the cohort to build more robust diagnostic models and refining the algorithms to make the MDC system portable. The ultimate goal is to enable depression risk analysis via mobile devices, providing timely warnings and support for individuals in their everyday environments.
Summary updated