SIGMA: speaker-isolated, symptom-structured multimodal estimation of depression severity from clinical interviews

Zhuoyuan Tang, Tao Wang, Kai Yang, Li Yuan, Yuezhou Zhang, Rongrong Zhong, Yi Cai, Saskia C. Sanderson, Jaime Delgadillo, Richard Dobson, Angus RobertsarXiv 2026 · arXiv

TL;DR

SIGMA is a new multimodal framework that achieves state-of-the-art results in estimating depression severity from clinical interviews. By isolating participant speech and using a "graded-response" head to model symptoms as related parts of a single latent trait, it significantly outperforms existing methods on the standard E-DAIC benchmark.

What Problem Does This Paper Address?

Automating depression assessment through clinical interviews is a high-stakes challenge. While machine learning models can analyze text, audio, and video, they often stumble over three hurdles:

  1. Interviewer Noise: Most models process the entire interview, meaning the interviewer’s questioning style and voice can "leak" into the data, acting as a confounding variable.
  2. Late Fusion: Many systems analyze each modality (text, sound, sight) separately and only combine them at the very end, missing the fine-grained interactions (e.g., how a hesitant tone changes the meaning of a specific word).
  3. Symptom Independence: Standard models often try to predict a total score or treat individual symptoms (like sleep issues vs. depressed mood) as completely independent tasks, ignoring the clinical reality that these symptoms are highly correlated.

Methods

The authors propose SIGMA (Speaker-Isolated Graded-response Multimodal Architecture), which consists of four key stages:

  • Participant-Speech Isolation: Using pyannote for speaker diarization, the model strips away the interviewer's turns, ensuring it only learns from the person being assessed.
  • Cross-Modal Fusion: Instead of simple concatenation, SIGMA uses six pairwise cross-attention modules. This allows the text, audio, and video streams to "condition" one another during the encoding process.
  • Graded-Response Head: Borrowing from psychometrics (Item Response Theory), the model maps the fused data to a single latent severity trait ($\theta$). This trait then predicts the eight PHQ-8 symptom items through ordered response curves, ensuring the model understands that "nearly every day" represents more severity than "several days."
  • Multi-task Learning: The model is trained jointly on all eight symptoms using a Weighted Imbalanced Ordinal Log-Loss (W-ImbOLL), which handles the fact that most people in the dataset report "not at all" for most symptoms.
The SIGMA pipeline: from raw interview to joint symptom prediction.

Datasets

The study uses the Extended Distress Analysis Interview Corpus (E-DAIC), a standard benchmark containing 275 semi-clinical interviews. These interviews involve a virtual agent ("Ellie") and are paired with PHQ-8 (Patient Health Questionnaire-8) scores.

Key Insights and Ideas

  • The Power of Isolation: Simply removing the interviewer's speech provided the largest performance gains. This suggests that interviewer-specific cues are a major source of "noise" in previous research.
  • Text is King (on this dataset): Surprisingly, the authors found that non-verbal streams (audio and video) contributed very little participant-specific evidence. In fact, they discovered a common pitfall: if a model isn't carefully "packed" to ignore padding, it might accidentally learn to predict depression based solely on interview length, which correlates with severity.
  • Affective vs. Somatic Symptoms: The model's graded-response head revealed that affective/cognitive symptoms (like feelings of failure or depressed mood) are much more "discriminative" in transcripts than somatic symptoms (like sleep or appetite changes).
  • Structured Outputs Matter: Modeling the ordinal nature of symptoms (the fact that scores 0, 1, 2, and 3 have a specific order and distance) is more effective than treating them as simple classification categories.
Fitted item parameters showing that affective symptoms (blue) have higher discriminative power than somatic ones (orange).

Key Results

SIGMA set a new state-of-the-art on the official E-DAIC test partition:

  • Concordance Correlation Coefficient (CCC): 0.762 (a 12.1% relative improvement over previous bests).
  • Root Mean Square Error (RMSE): 4.13 (a 7.6% reduction in error).
  • Mean Absolute Error (MAE): 3.30 (a 12.2% reduction in error).

The model also performs well as a screening tool, achieving an AUC of 0.89 for identifying individuals scoring above the clinical cutoff (PHQ-8 $\ge$ 10).

Predicted vs. observed PHQ-8 scores, showing strong correlation but some compression toward the mean.

Limitations and Future Work

  • Dataset Constraints: E-DAIC is relatively small (56 test participants) and limited to English-speaking adults in the US. Generalizability to other cultures or languages is unknown.
  • Diarization Heuristics: The model assumes the participant is the "dominant speaker," which might fail in interviews where the interviewer is particularly talkative.
  • Non-Verbal Features: The lack of contribution from audio/video might be a limitation of the pre-extracted features provided with the dataset rather than a lack of signal in those modalities themselves. Future work could explore end-to-end raw signal processing.
  • Clinical Use: The authors emphasize that this is a research framework for screening support, not a standalone diagnostic system.

Summary updated

Abstract

Clinical interviews are central to depression assessment but time-consuming and rater-dependent. Automated severity estimation offers scalable screening, yet existing methods often retain interviewer speech (confounding participant cues), fuse modalities late (losing fine-grained cross-modal dynamics), or score symptoms independently (ignoring inter-symptom correlations). We present SIGMA, a speaker-aware, symptom-structured framework for depression-severity estimation from semi-clinical interviews. SIGMA uses speaker diarization to restrict input to participant speech, cross-attention so that text, audio and video condition one another during encoding, and a graded-response head that predicts the eight questionnaire items jointly through a shared latent severity trait with item-specific ordered thresholds. On the Extended Distress Analysis Interview Corpus, SIGMA achieves state-of-the-art performance (CCC 0.762, RMSE 4.13, MAE 3.30 over five seeds). Ablations confirm that isolating participant speech and joint, structured symptom prediction each substantially improve concordance and estimation accuracy.
BibTeX
@misc{tang2026sigma,
  author = {Zhuoyuan Tang and Tao Wang and Kai Yang and Li Yuan and Yuezhou Zhang and Rongrong Zhong and Yi Cai and Saskia C. Sanderson and Jaime Delgadillo and Richard Dobson and Angus Roberts},
  title = {{SIGMA: speaker-isolated, symptom-structured multimodal estimation of depression severity from clinical interviews}},
  year = {2026},
  eprint = {2611.23348},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23348},
}
Figure