TL;DR
Incorporating therapist prompts into automatic depression detection models can artificially inflate performance due to a hidden structural bias in clinical interviews. Instead of learning to analyze patient behavior, models learn to exploit specific follow-up questions as "discriminative shortcuts," achieving a misleadingly high but brittle 0.90 F1 score.
What Problem Does This Paper Address?
Automatic depression detection from text transcripts typically relies on analyzing a patient's language. Researchers frequently use the Distress Analysis Interview Corpus - Wizard of Oz (DAIC-WOZ) dataset, which contains clinical interviews conducted by a virtual agent named Ellie.
Recently, several state-of-the-art models reported massive performance boosts by including Ellie’s questions alongside the participant's answers, claiming the prompts provide crucial "supplementary context." This paper challenges that assumption. The authors hypothesize that this improvement isn't due to superior model architectures, but rather a systemic bias within the interviewer's prompts that allows models to cheat.
Methods
To uncover this potential bias, the authors designed a rigorous ablation study using two distinct model architectures:
- LongBERT: A BERT-based Longformer model optimized for processing long documents that exceed the standard 512-token limit.
- Graph Convolutional Network (GCN): A sequence-agnostic model that maps both words and participants into a shared two-dimensional output space, making its internal representations highly interpretable.
The authors trained variants of these models on three configurations:
- Participant-only (P): Using only the patient's responses.
- Interviewer-only (E): Using only Ellie's prompts.
- Ensemble ($P \wedge E$): A simple voting ensemble where a participant is classified as depressed only if both the P-variant and E-variant agree.
To understand what the models were learning, they extracted the "keywords" the GCN associated with the depression class and plotted their distribution across the timeline of each interview.
Datasets
The study exclusively utilizes the DAIC-WOZ dataset, a widely recognized multimodal benchmark for depression classification. It consists of:
- Training set: 107 files (77 control, 30 depressed)
- Evaluation set: 35 files (23 control, 12 depressed)
The dataset includes audio, video, and text transcripts of semi-structured interviews led by the virtual interviewer, Ellie, who draws from a finite repertoire of 191 prompts.
Key Insights and Ideas
- The Interviewer Gives It Away: Shockingly, models trained solely on the interviewer's questions performed better than those trained on the actual patients' answers.
- The Mechanics of the Bias: Ellie follows a semi-structured decision tree. If a participant answers affirmatively to a screening question (e.g., "Have you been diagnosed with depression?"), Ellie is programmed to follow up with specific, deeper questions (e.g., "How long ago?", "What got you to seek help?"). The machine learning models simply learned to recognize these specific follow-up prompts as a proxy for a depression diagnosis.
- Localization vs. Generalization: As visualized in the paper's keyword heatmaps, the participant-only model (P-GCN) gathers subtle evidence scattered across the entire interview. In contrast, the interviewer-only model (E-GCN) focuses narrowly on a single segment just past the halfway mark where these personal health questions occur.
- An Illustrative Example: In specific interview segments, the model completely ignores the patient's text and relies entirely on the presence of Ellie's targeted probing to make a perfect classification.
Key Results
The quantitative results clearly demonstrate the strength of the interviewer bias:
- Participant-only GCN (P-GCN) achieved a strong baseline macro $F_1$ score of 0.85.
- Interviewer-only GCN (E-GCN) bypassed the patient entirely and achieved an $F_1$ score of 0.88.
- The Exploitative Ensemble ($P\text{-GCN} \wedge E\text{-GCN}$) intentionally combined both sources to exploit the bias, yielding a remarkable 0.90 macro $F_1$ score—the highest text-only result reported on this dataset to date.
- Temporal Ablation: Training
E-longBERTon just the second half of the interviews yielded an $F_1$ of 0.84 (identical to using the full interview), whereas training it on the first half caused performance to plummet to 0.60, confirming the bias is heavily concentrated in the latter half of the sessions.
Limitations and Future Work
- Clinical Setup Constraint: This study focuses strictly on controlled, semi-structured clinical interviews. These findings do not apply to "wild settings" like social media text analysis where no interviewer dictates the flow.
- Text-Only Modality: The analysis was restricted to textual transcripts. Because Ellie's acoustic features and gestures are also present in the multimodal data, the authors note that future work must investigate whether similar biases are being exploited in audio and video models.
- Dataset Specificity: The exact shortcut identified is unique to the DAIC-WOZ protocol. The authors plan to replicate this analysis on other datasets (like E-DAIC, EATD, and ANDROIDS) to see if similar interviewer-induced artifacts exist elsewhere.
Summary updated