E-Prevention: Advanced Support System for Monitoring and Relapse Prevention in Patients with Psychotic Disorders Analyzing Long-Term Multimodal Data from Wearables and Video Captures

Athanasia Zlatintsi, Panagiotis Paraskevas Filntisis, Christos Garoufis, Niki Efthymiou, Petros Maragos, Andreas Menychtas, Ilias Maglogiannis, Panayiotis Tsanakas, Thomas Sounapoglou, Emmanouil Kalisperakis, Thomas Karantinos, Marina Lazaridi, Vasiliki Garyfalli, Asimakis Mantas, Leonidas Mantonakis, Nikolaos Smyrnis

TL;DR

This paper introduces "e-Prevention," an integrated system using wearable sensors and video analysis to monitor patients with psychotic disorders and predict relapses. By analyzing long-term multimodal data, the system aims to revolutionize psychiatric care by enabling proactive intervention and improving patient outcomes.

What Problem Does This Paper Address?

Psychotic disorders, such as schizophrenia and bipolar disorder, are characterized by unpredictable relapses that significantly impact patients' lives. Current methods for monitoring and predicting these relapses are often subjective, infrequent, and rely on self-reporting, which can be unreliable. The paper addresses the critical need for objective, continuous, and long-term monitoring to identify early warning signs of relapse and enable timely intervention, thereby improving patient well-being and reducing the catastrophic effects of relapses.

Methods

The e-Prevention system employs a multi-pronged approach:

  • Long-term Biosignal Monitoring: A smartwatch continuously records biometric and behavioral data (accelerometer, gyroscope, heart rate, heart rate variability) over extended periods.
  • Video-based Clinical Interviews: A tablet captures video recordings of patients during interviews with clinicians, allowing for the analysis of speech and facial expressions.
  • Cloud-based Data Management: All collected data are systematically stored and managed on a dedicated cloud server.
  • Machine and Deep Learning for Relapse Detection:
    • Physiological Data Analysis: Autoencoder architectures (Transformer, FNN, CNN, GRU) are used for anomaly detection to identify patterns indicative of relapses from smartwatch data. Both personalized (per-patient) and global (patient-independent) models are explored.
  • Speech and Facial Expression Analysis: Convolutional Autoencoders (CAEs) and Convolutional Variational Autoencoders (CVAEs) are applied to spontaneous speech and facial cues from video interviews to detect and predict relapses.
  • Multimodal Fusion: The system investigates combining audio and physiological data to enhance relapse detection accuracy.
  • Statistical Analysis: Extensive statistical analysis is performed to identify significant differences in features between healthy controls and patients, paving the way for biomarker discovery.

Datasets

The study utilizes data collected from participants in the e-Prevention project. This includes:

  • Smartwatch Data: Long-term (up to one year) continuous data from accelerometers, gyroscopes, and PPG sensors from both healthy controls and patients with psychotic disorders (schizophrenia, bipolar disorder).
  • Video Interview Data: Audio-visual recordings of interviews between patients and clinicians.
  • Demographic and Clinical Data: Information such as age, education, illness duration, and psychopathological scales (PANSS, WHODAS II).

The paper does not explicitly mention newly introduced public benchmarks but focuses on the novel integrated system and its application to collected data.

Key Insights and Ideas

  • Multimodal Data is Key: Combining physiological data from wearables with audio-visual information from interviews provides a richer, more robust picture of a patient's state than any single modality alone.
  • Long-term Monitoring is Crucial: Capturing data over extended periods (months to a year) is essential for identifying subtle changes and patterns that precede relapses, which short-term studies might miss.
  • Personalized Models Outperform Global Models (with caveats): While personalized autoencoder models generally achieve better relapse detection performance, the research explores global models to assess their scalability and generalizability.
  • Speech and Facial Cues are Informative: Spontaneous speech and facial expressions contain valuable information for detecting and predicting relapses, especially when analyzed with advanced deep learning techniques.
  • Relapse Severity Matters: The severity of a relapse appears to influence the anomaly scores, with moderate and severe relapses yielding better detection results.
  • Per-Patient Normalization is Important: Normalizing data on a per-patient basis significantly improves the performance of global models, suggesting that individual baselines are critical.

Key Results

  • Physiological Data: Statistical analysis revealed significant differences in various features (e.g., movement variability, HRV nonlinear metrics) between controls and patients. Autoencoder models achieved promising results in relapse detection, with personalized CNN AE models showing a PR-AUC of up to 0.97 for some patients.
  • Speech Data: CAE and CVAE models demonstrated effectiveness in detecting and predicting relapses from spontaneous speech, with CVAEs showing promise in patient-independent settings.
  • Facial Cues: Facial expressions were found to correlate with psychopathological scales, suggesting their potential for automated assessment of symptom severity.
  • Multimodal Fusion: Combining audio and physiological data yielded higher ROC-AUC scores than using single modalities, indicating the value of auxiliary information. Additive fusion showed slightly improved results over multiplicative fusion.

Limitations and Future Work

  • Data Availability: The paper notes that data availability is restricted due to privacy concerns and the ongoing nature of the project.
  • Relapse Data Scarcity: For some patients, the amount of available relapse data was limited, potentially affecting model performance. The authors acknowledge that this can lead to lower performance for certain patients.
  • Generalizability of Global Models: While global models were explored, their performance was generally lower than personalized models, especially when dealing with low-severity relapses.
  • Future Work: The authors suggest exploring the use of more advanced fusion techniques, investigating the impact of different temporal aggregation methods, and further refining models for patient-independent relapse prediction. The potential for using these markers to predict psychopathology and prevent relapses is highlighted as a key future direction.

Summary updated

Abstract

Wearable technologies and digital phenotyping foster unique opportunities for designing novel intelligent electronic services that can address various well-being issues in patients with mental disorders (i.e., schizophrenia and bipolar disorder), thus having the potential to revolutionize psychiatry and its clinical practice. In this paper, we present e-Prevention, an innovative integrated system for medical support that facilitates effective monitoring and relapse prevention in patients with mental disorders. The technologies offered through e-Prevention include: (i) long-term continuous recording of biometric and behavioral indices through a smartwatch; (ii) video recordings of patients while being interviewed by a clinician, using a tablet; (iii) automatic and systematic storage of these data in a dedicated Cloud server and; (iv) the ability of relapse detection and prediction. This paper focuses on the description of the e-Prevention system and the methodologies developed for the identification of feature representations that correlate with and can predict psychopathology and relapses in patients with mental disorders. Specifically, we tackle the problem of relapse detection and prediction using Machine and Deep Learning techniques on all collected data. The results are promising, indicating that such predictions could be made and leading eventually to the prediction of psychopathology and the prevention of relapses.
BibTeX
@inproceedings{zlatintsi2022e,
  author = {Athanasia Zlatintsi and Panagiotis Paraskevas Filntisis and Christos Garoufis and Niki Efthymiou and Petros Maragos and Andreas Menychtas and Ilias Maglogiannis and Panayiotis Tsanakas and Thomas Sounapoglou and Emmanouil Kalisperakis and Thomas Karantinos and Marina Lazaridi and Vasiliki Garyfalli and Asimakis Mantas and Leonidas Mantonakis and Nikolaos Smyrnis},
  title = {{E-Prevention: Advanced Support System for Monitoring and Relapse Prevention in Patients with Psychotic Disorders Analyzing Long-Term Multimodal Data from Wearables and Video Captures}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=nURuZOKTTf},
}

Loading related content...

Figure