DeepTMH: Multimodal Semi-supervised framework leveraging Affective and Cognitive engagement for Telemental Health.

Pooja Guhan, Naman Awasthi, Ritwika Das, Manas Agarwal, Kathryn McDonald, Kristin Bussell, Dinesh Manocha, Gloria Reeves, Aniket BeraarXiv 2021 · arXiv

Abstract

To aid existing telemental health services, we propose DeepTMH, a novel framework that models telemental health session videos by extracting latent vectors corresponding to Affective and Cognitive features frequently used in psychology literature. Our approach leverages advances in semi-supervised learning to tackle the data scarcity in the telemental health session video domain and consists of a multimodal semi-supervised GAN to detect important mental health indicators during telemental health sessions. We demonstrate the usefulness of our framework and contrast against existing works in two tasks: Engagement regression and Valence-Arousal regression, both of which are important to psychologists during a telemental health session. Our framework reports 40% improvement in RMSE over SOTA method in Engagement Regression and 50% improvement in RMSE over SOTA method in Valence-Arousal Regression. To tackle the scarcity of publicly available datasets in telemental health space, we release a new dataset, MEDICA, for mental health patient engagement detection. Our dataset, MEDICA consists of 1299 videos, each 3 seconds long. To the best of our knowledge, our approach is the first method to model telemental health session data based on psychology-driven Affective and Cognitive features, which also accounts for data sparsity by leveraging a semi-supervised setup.

Introduction

Telehealth has become a vital tool for mental healthcare, but it strips away the subtle non-verbal cues—like body posture, handshakes, or shared physical space—that psychotherapists traditionally use to gauge a patient's "therapeutic alliance" or engagement. Currently, there are no objective, real-time measurements for this. Therapists must rely on self-reports (which are prone to bias) or "show rates" (which are lagging indicators). Furthermore, training for telehealth-specific engagement is limited.

The DeepTMH framework aims to bridge this gap by providing an automated tool that can assist psychotherapists in diagnosing disorders and improving patient retention by monitoring engagement levels through a standard webcam and microphone.

Methodology

The DeepTMH framework is built on a multi-componential approach that mimics how a human therapist might evaluate a patient. It integrates three primary pillars:

  1. Cognitive State Mode: This focuses on audio features linked to cognitive processing and stress. It extracts four specific types of features: Glottal (vocal cord tension), Prosody (intonation/loudness), Phonation (stability of vibration), and Articulation (movement of lips/tongue).
  2. Affective State Mode: This looks for emotional consistency across three modalities:
  3. Visuals: Using a VGG-B architecture to extract facial expressions.
  4. Audio: Using Mel-frequency cepstrum (MFCC) features for emotion in speech.
  5. Text: Using a BERT-based model to analyze the sentiment of what is being said.
  6. Semi-Supervised GAN (Generative Adversarial Network): High-quality labeled medical data is notoriously hard to get. To solve this, the authors used a GAN where a Generator creates "fake" feature maps and a Discriminator learns to distinguish between real (labeled and unlabeled) and fake data. This allows the model to learn from a massive amount of unlabeled video data, making it more robust and generalizable.
The DeepTMH architecture combining Cognitive and Affective feature extraction with a semi-supervised GAN.

Datasets

To train and validate the framework, the researchers utilized a combination of new and existing datasets:

  • MEDICA (New): Introduced specifically for telemental health, this dataset contains 1,299 video clips (3 seconds each) derived from 13 mock therapy sessions. It is labeled for engagement, stress, attentiveness, and eight different emotions.
  • CMU-MOSEI: A large-scale dataset of sentiment and emotion used as the source for 12,854 unlabeled data points to train the semi-supervised GAN.
  • Real-World Data: The authors collaborated with 8 child psychotherapists and 20 caregivers to record and evaluate actual telemental sessions to validate the model's real-world utility.
Sample video frames representing the diverse subjects and settings in telemental health sessions.

Key Results and Insights

DeepTMH demonstrates significant performance gains over existing methods:

  • Superior Accuracy: The framework achieved a Root Mean Squared Error (RMSE) of 0.10 on the MEDICA dataset, a 40% improvement over the next best method (S3VM at 0.17) and a 50% improvement in Valence-Arousal Regression.
  • Ablation Success: Testing showed that combining both Affective (A) and Cognitive (C) features yielded the best results (0.10 RMSE) compared to using them individually (0.24 and 0.30 respectively).
  • Clinical Correlation: In real-world tests, DeepTMH’s engagement scores showed a positive Pearson correlation (0.38 - 0.40) with the Working Alliance Inventory (WAI) scores—the gold standard used by therapists to measure the therapeutic bond.

The study proves that psychology-informed ML—targeting features like glottal tension and emotional incongruity—outperforms "black-box" models. Furthermore, the semi-supervised setup effectively addresses the scarcity of labeled clinical data.

Limitations and Future Work

While promising, several hurdles remain:

  • Technical Constraints: Performance can degrade due to video occlusions, missing modalities, or data corruption caused by low internet bandwidth.
  • Diversity: Future work aims to expand the MEDICA dataset to include more cultural and geographical variations to ensure inclusivity.
  • Explainability: A major goal is to make the AI’s decisions explainable, providing therapists with insights into why a patient is perceived as disengaged (e.g., "detected high vocal tension").

Summary updated

Abstract

To aid existing telemental health services, we propose DeepTMH, a novel framework that models telemental health session videos by extracting latent vectors corresponding to Affective and Cognitive features frequently used in psychology literature. Our approach leverages advances in semi-supervised learning to tackle the data scarcity in the telemental health session video domain and consists of a multimodal semi-supervised GAN to detect important mental health indicators during telemental health sessions. We demonstrate the usefulness of our framework and contrast against existing works in two tasks: Engagement regression and Valence-Arousal regression, both of which are important to psychologists during a telemental health session. Our framework reports 40% improvement in RMSE over SOTA method in Engagement Regression and 50% improvement in RMSE over SOTA method in Valence-Arousal Regression. To tackle the scarcity of publicly available datasets in telemental health space, we release a new dataset, MEDICA, for mental health patient engagement detection. Our dataset, MEDICA consists of 1299 videos, each 3 seconds long. To the best of our knowledge, our approach is the first method to model telemental health session data based on psychology-driven Affective and Cognitive features, which also accounts for data sparsity by leveraging a semi-supervised setup.
BibTeX
@misc{guhan2021deeptmh,
  author = {Pooja Guhan and Naman Awasthi and Ritwika Das and Manas Agarwal and Kathryn McDonald and Kristin Bussell and Dinesh Manocha and Gloria Reeves and Aniket Bera},
  title = {{DeepTMH: Multimodal Semi-supervised framework leveraging Affective and Cognitive engagement for Telemental Health.}},
  year = {2021},
  eprint = {2011.08690},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.08690},
}
Figure