DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews

Sergio Burdisso, Ernesto Reyes-Ramírez, Esaú Villatoro-Tello, Fernando Sánchez-Vega, Pastor López-Monroy, Petr Motlicek

TL;DR

Incorporating therapist prompts into automatic depression detection models can artificially inflate performance due to a hidden structural bias in clinical interviews. Instead of learning to analyze patient behavior, models learn to exploit specific follow-up questions as "discriminative shortcuts," achieving a misleadingly high but brittle 0.90 F1 score.

What Problem Does This Paper Address?

Automatic depression detection from text transcripts typically relies on analyzing a patient's language. Researchers frequently use the Distress Analysis Interview Corpus - Wizard of Oz (DAIC-WOZ) dataset, which contains clinical interviews conducted by a virtual agent named Ellie.

Recently, several state-of-the-art models reported massive performance boosts by including Ellie’s questions alongside the participant's answers, claiming the prompts provide crucial "supplementary context." This paper challenges that assumption. The authors hypothesize that this improvement isn't due to superior model architectures, but rather a systemic bias within the interviewer's prompts that allows models to cheat.

Methods

To uncover this potential bias, the authors designed a rigorous ablation study using two distinct model architectures:

  1. LongBERT: A BERT-based Longformer model optimized for processing long documents that exceed the standard 512-token limit.
  2. Graph Convolutional Network (GCN): A sequence-agnostic model that maps both words and participants into a shared two-dimensional output space, making its internal representations highly interpretable.

The authors trained variants of these models on three configurations:

  • Participant-only (P): Using only the patient's responses.
  • Interviewer-only (E): Using only Ellie's prompts.
  • Ensemble ($P \wedge E$): A simple voting ensemble where a participant is classified as depressed only if both the P-variant and E-variant agree.

To understand what the models were learning, they extracted the "keywords" the GCN associated with the depression class and plotted their distribution across the timeline of each interview.

Datasets

The study exclusively utilizes the DAIC-WOZ dataset, a widely recognized multimodal benchmark for depression classification. It consists of:

  • Training set: 107 files (77 control, 30 depressed)
  • Evaluation set: 35 files (23 control, 12 depressed)

The dataset includes audio, video, and text transcripts of semi-structured interviews led by the virtual interviewer, Ellie, who draws from a finite repertoire of 191 prompts.

Key Insights and Ideas

  • The Interviewer Gives It Away: Shockingly, models trained solely on the interviewer's questions performed better than those trained on the actual patients' answers.
  • The Mechanics of the Bias: Ellie follows a semi-structured decision tree. If a participant answers affirmatively to a screening question (e.g., "Have you been diagnosed with depression?"), Ellie is programmed to follow up with specific, deeper questions (e.g., "How long ago?", "What got you to seek help?"). The machine learning models simply learned to recognize these specific follow-up prompts as a proxy for a depression diagnosis.
  • Localization vs. Generalization: As visualized in the paper's keyword heatmaps, the participant-only model (P-GCN) gathers subtle evidence scattered across the entire interview. In contrast, the interviewer-only model (E-GCN) focuses narrowly on a single segment just past the halfway mark where these personal health questions occur.
Figure 1: Heatmaps illustrating the distribution of learned keywords by each model across the progression of each interview. The x-axis represents individual interviews, while the y-axis denotes the percentage of the conversation from the beginning (0%) to the end (100%). The white vertical line in each plot indicates the training and evaluation splits respectively. Finally, in the E-GCN evaluation split region, the small red rectangle depicts the interview segment showed in Fig. 2.
  • An Illustrative Example: In specific interview segments, the model completely ignores the patient's text and relies entirely on the presence of Ellie's targeted probing to make a perfect classification.
Figure 2: Illustrative segment from interview "381" in the evaluation set, highlighted in Figure 1. Conversation turns are color-coded based on the proportion of key- words present, with keywords underlined for emphasis.

Key Results

The quantitative results clearly demonstrate the strength of the interviewer bias:

  • Participant-only GCN (P-GCN) achieved a strong baseline macro $F_1$ score of 0.85.
  • Interviewer-only GCN (E-GCN) bypassed the patient entirely and achieved an $F_1$ score of 0.88.
  • The Exploitative Ensemble ($P\text{-GCN} \wedge E\text{-GCN}$) intentionally combined both sources to exploit the bias, yielding a remarkable 0.90 macro $F_1$ score—the highest text-only result reported on this dataset to date.
  • Temporal Ablation: Training E-longBERT on just the second half of the interviews yielded an $F_1$ of 0.84 (identical to using the full interview), whereas training it on the first half caused performance to plummet to 0.60, confirming the bias is heavily concentrated in the latter half of the sessions.

Limitations and Future Work

  • Clinical Setup Constraint: This study focuses strictly on controlled, semi-structured clinical interviews. These findings do not apply to "wild settings" like social media text analysis where no interviewer dictates the flow.
  • Text-Only Modality: The analysis was restricted to textual transcripts. Because Ellie's acoustic features and gestures are also present in the multimodal data, the authors note that future work must investigate whether similar biases are being exploited in audio and video models.
  • Dataset Specificity: The exact shortcut identified is unique to the DAIC-WOZ protocol. The authors plan to replicate this analysis on other datasets (like E-DAIC, EATD, and ANDROIDS) to see if similar interviewer-induced artifacts exist elsewhere.

Summary updated

Abstract

Automatic depression detection from conversational data has gained significant interest in recent years. The DAIC-WOZ dataset, interviews conducted by a human-controlled virtual agent, has been widely used for this task. Recent studies have reported enhanced performance when incorporating interviewer's prompts into the model. In this work, we hypothesize that this improvement might be mainly due to a bias present in these prompts, rather than the proposed architectures and methods. Through ablation experiments and qualitative analysis, we discover that models using interviewer's prompts learn to focus on a specific region of the interviews, where questions about past experiences with mental health issues are asked, and use them as discriminative shortcuts to detect depressed participants. In contrast, models using participant responses gather evidence from across the entire interview. Finally, to highlight the magnitude of this bias, we achieve a 0.90 F1 score by intentionally exploiting it, the highest result reported to date on this dataset using only textual information. Our findings underline the need for caution when incorporating interviewers' prompts into models, as they may inadvertently learn to exploit targeted prompts, rather than learning to characterize the language and behavior that are genuinely indicative of the patient's mental health condition.
BibTeX
@inproceedings{burdisso2024daic,
  author = {Sergio Burdisso and Ernesto Reyes-Ramírez and Esaú Villatoro-Tello and Fernando Sánchez-Vega and Pastor López-Monroy and Petr Motlicek},
  title = {{DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}

Loading related content...

Figure