DAIC-WOZ: On the Validity of Using the Therapist’s prompts in Automatic Depression Detection from Clinical Interviews

Sergio Burdisso, Ernesto Reyes-Ramírez, Esaú Villatoro-tello, Fernando Sánchez-Vega, Adrian Lopez Monroy, Petr Motlicek

TL;DR

This paper investigates the effectiveness of using interviewer prompts in automatic depression detection from clinical interviews, finding that models achieve artificially inflated performance by exploiting biases in these prompts rather than learning genuine indicators of depression. The authors demonstrate that by focusing on specific question segments, models can achieve state-of-the-art results, highlighting the need for caution in applying such methods to real-world clinical settings.

What Problem Does This Paper Address?

The automatic detection of depression from clinical interviews is a growing area of AI research, aiming to improve the accessibility and standardization of mental healthcare. The DAIC-WOZ dataset, featuring interviews conducted by a virtual interviewer named Ellie, is a popular benchmark. While recent studies have shown improved performance by incorporating Ellie's prompts alongside participant responses, this paper questions whether this improvement is due to genuine learning or an exploitable bias within Ellie's prompts. The core problem is to determine if current models are truly learning to detect depression or if they are merely capitalizing on a shortcut provided by the interviewer's specific questioning patterns, which may not generalize to real-world scenarios.

Methods

The authors employ a combination of ablation studies and qualitative analysis to dissect the contribution of interviewer prompts. They train and evaluate two main models: a Longformer-based BERT model and a Graph Convolutional Network (GCN). For each model, they compare performance using only participant responses (P-model) versus using only Ellie's prompts (E-model). The GCN model is particularly useful due to its interpretability, allowing the researchers to extract learned "keywords" and analyze their distribution throughout the interviews. This analysis helps visualize where each model focuses its attention.

Datasets

The primary dataset used is the DAIC-WOZ dataset. This dataset consists of semi-structured clinical interviews conducted by a virtual interviewer (Ellie) with participants, some of whom are diagnosed with depression. The dataset includes transcribed text, audio, and video, though this study focuses solely on the textual modality.

Key Insights and Ideas

  • Prompt Bias is a Major Factor: The most significant finding is that interviewer prompts in the DAIC-WOZ dataset contain a bias that models can easily exploit. This bias leads to artificially high performance metrics.
  • Targeted Questioning as a Shortcut: Models trained with Ellie's prompts tend to focus on very specific segments of the interview, particularly later parts where more personal questions about mental health are asked. This is a "discriminative shortcut" rather than a holistic understanding of the participant's condition.
  • Participant Responses Offer Broader Evidence: In contrast, models using only participant responses gather evidence from across the entire interview, suggesting a more generalized approach to depression detection.
  • Interpretability Reveals the Bias: The GCN model's ability to identify learned keywords and their distribution was crucial in demonstrating that Ellie's model concentrates on localized question clusters, while the participant model is more distributed.
  • Ensembling Amplifies Bias: Combining participant and interviewer data through ensembling further boosts performance, indicating that the bias in prompts can be effectively leveraged when combined with participant data.

Key Results

  • State-of-the-Art Performance via Bias Exploitation: By intentionally exploiting the bias in Ellie's prompts, the authors achieved an F1 score of 0.90, which is the highest reported score to date on the DAIC-WOZ dataset using only textual information.
  • E-GCN Outperforms Previous Works: The E-GCN model, using only Ellie's prompts, achieved an F1 score of 0.88, outperforming most previous works that relied solely on textual input.
  • Significant Improvement from Prompts: For the Longformer models, the F1 score for the depression group improved from 0.64 (participant only) to 0.80 when Ellie's prompts were included.
  • Localized Focus of E-GCN: Heatmaps clearly showed that the E-GCN model concentrated its attention on specific interview segments, particularly in the latter half, whereas the P-GCN model's focus was more distributed.

Limitations and Future Work

  • Task Configuration: The study is limited to the specific task of depression detection within a controlled clinical interview setting ("wild setting" analysis, like social media, is not covered).
  • Modality Specificity: The research focused solely on textual data. Future work aims to investigate if similar biases exist in acoustic modalities and to extend the analysis to other modalities present in the DAIC-WOZ corpus.
  • Dataset Specificity: The findings are specific to the DAIC-WOZ corpus. The authors plan to replicate their analysis on other datasets (e.g., E-DAIC, EATD, ANDROIDS) to confirm the generalizability of their findings.
  • Future Prompt Generation: Future work includes validating the impact of prompts generated by a fully automatic therapist in similar setups and corpora.

Summary updated

Abstract

Automatic depression detection from conversational data has gained significant interest in recent years. The DAIC-WOZ dataset, interviews conducted by a human-controlled virtual agent, has been widely used for this task. Recent studies have reported enhanced performance when incorporating interviewer's prompts into the model. In this work, we hypothesize that this improvement might be mainly due to a bias present in these prompts, rather than the proposed architectures and methods. Through ablation experiments and qualitative analysis, we discover that models using interviewer's prompts learn to focus on a specific region of the interviews, where questions about past experiences with mental health issues are asked, and use them as discriminative shortcuts to detect depressed participants. In contrast, models using participant responses gather evidence from across the entire interview. Finally, to highlight the magnitude of this bias, we achieve a 0.90 F1 score by intentionally exploiting it, the highest result reported to date on this dataset using only textual information. Our findings underline the need for caution when incorporating interviewers' prompts into models, as they may inadvertently learn to exploit targeted prompts, rather than learning to characterize the language and behavior that are genuinely indicative of the patient's mental health condition.
BibTeX
@inproceedings{burdisso2024daic,
  author = {Sergio Burdisso and Ernesto Reyes-Ramírez and Esaú Villatoro-tello and Fernando Sánchez-Vega and Adrian Lopez Monroy and Petr Motlicek},
  title = {{DAIC-WOZ: On the Validity of Using the Therapist’s prompts in Automatic Depression Detection from Clinical Interviews}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure