Several recent systems ask a large language model (LLM) to screen for depression from a clinical interview and to write a short rationale alongside the severity score, for example "the patient speaks slowly, with flat pitch, and reports loss of interest." Clinicians value such rationales because a bare number from a black-box classifier is hard to trust. The rationale is only useful, though, if it describes this patient. An LLM that has absorbed the textbook picture of depression can write a fluent, plausible paragraph about blunted affect and long pauses whether or not the recording contains them. That is a hallucination, in the sense of fluent but ungrounded text, and in this setting it is invisible: in a single recording, a correct rationale and a stereotyped one read the same.
This note proposes a way to catch it. Depression research has spent two decades building objective speech measurements to get around the subjectivity of questionnaires and interviews: pitch variability, formant structure, energy, and response latency. Those same measurements can serve as ground truth for the rationale's claims about how the patient sounds. We take real interview recordings and transplant one participant's acoustic evidence into another participant's session. We then ask whether the model's score and its stated observations follow the transplanted evidence or stay anchored to the transcript, the interviewer's questions, and the model's stereotype of depression. The deliverable is an audit protocol and a benchmark of counterfactual sessions built from existing corpora. It also includes a head-to-head test against a much simpler system that turns measured features into templated sentences. That test decides whether free-text LLM rationales earn their risk at all.
Core Claim
Existing faithfulness tests for clinical LLMs remove evidence and see what happens. That design cannot tell "the model reads the specific values" apart from "the model notices that some evidence is present and fills in the expected story." Swapping evidence between real patients with different severities can tell them apart. We claim three things. First, when behavioral speech features are swapped across participants, the rationales of current multimodal depression LLMs will mostly keep describing the original participant's presumed symptoms. Second, many of their paralinguistic claims will be false when checked against measured features. Third, the claims that do flip under the swap will line up with the interviewer's question sequence, a known shortcut, more than with the patient's own evidence. If the reverse holds and rationales do follow transplanted evidence, that is also worth knowing, and it would be the first positive faithfulness result in psychiatric screening.
Relation to Prior Work
The closest method is The Hallucination Dependence Index. It compares a model's unsupported-claim rate with and without a patient evidence bundle on 119 lung-cancer cases. It found that the model with the highest grounded support score (93.2%) was not the most faithful once prior reliance was taken into account, and HDI ranged from 0.336 to 0.984 across four models. That paper motivates our design and supplies half of it: we reuse its idea that grounding must be measured relative to what the model says without evidence. We depart from it in two ways. Its intervention is removal, while ours is cross-patient substitution. And its claims were adjudicated by experts, while many of our paralinguistic claims can be checked mechanically against acoustic measurements. It also never touched psychiatry or speech.
The systems we audit come from a small cluster. MLlm-DR maps pre-extracted audio and visual features into a frozen LLM through a small query transformer and trains it jointly to regress a PHQ-8 score and generate a rationale. Its authors themselves flag the risk that rationales contain fabricated observations, and they report expert agreement with rationales rather than a check of the rationales against the features. Predicting Depression in Screening Interviews from Interactive Multi-Theme Collaboration (PDIMC) summarizes transcripts by theme with Qwen2.5 before classification. InterMind adds retrieval over the DSM-5 and chain-of-thought prompting to text-only dialogues. None of the three measures whether its generated text is conditioned on the individual patient. We treat them as subjects of the audit rather than as competitors.
Two results constrain the design. DAIC-WOZ: On the Validity of Using the Therapist's prompts showed that a model reading only the virtual interviewer's prompts reaches F1 0.88, while a participant-only Longformer reaches 0.64. Any audit on this corpus must therefore control for the interviewer channel, or it will mistake shortcut-following for grounding. CounselBench found that an LLM judge detected clinical failure modes at only F1 0.5. That rules out using an LLM as the verifier of rationale claims, and it is why we tie verification to measured signals and human annotation.
The acoustic literature supplies the verifier. Hewlett Sanchez et al. and Cummins et al. establish which features separate depressed from control speech. They also show that these features are tangled with speaker identity: MFCC accuracy fell from 80% to 65% under speaker normalization. FairReFuse shows that gender confounds acoustic screening. Both findings shape how we pair swap donors.
Research Program
The benchmark is built on E-DAIC, which has PHQ-8 labels, audio, transcripts with timestamps, and pre-extracted features, and on DAIC-WOZ for interviewer-channel analyses. For each test session we build four variants. The original is left intact. In the acoustic swap, the pre-extracted acoustic feature stream is replaced by that of a donor participant whose PHQ-8 differs by at least a severity tier. The swap is aligned by interview question, so each answer's features come from the donor's answer to the same prompt. In the interviewer-only variant, participant turns are removed. In the no-acoustics variant, the acoustic stream is withheld. Donors are matched on gender by default, so a swap cannot be explained as a gender change. A separate cross-gender arm measures the confound directly. For text-and-audio models we also run the mirror swap, which transplants the donor's transcript while keeping the recipient's audio. Where MLlm-DR accepts only pre-extracted features, the swap happens at feature level, which avoids synthesizing any audio.
Each rationale is split into atomic claims, and each claim is labeled as paralinguistic (pitch, energy, speech rate, pause or response latency), transcript-content ("reports poor sleep"), or generic ("symptoms consistent with depression"). Paralinguistic claims are checked against measurements computed from the audio actually supplied to the model. For example, "reduced pitch variability" counts as supported if the session's F0 standard deviation falls in the bottom tercile of same-gender E-DAIC speakers, and "long pauses" is checked against timestamp-derived response latency. The tercile thresholds are fixed before any model output is seen. Transcript-content claims are checked by two trained annotators against the transcript. A 300-claim subset of paralinguistic claims is also judged by annotators listening to the audio, which estimates how often the automatic verifier disagrees with humans.
The central quantity is a tracking ratio for each output channel:
$$T = \frac{\mathbb{E}[\,y(\text{recipient with donor evidence}) - y(\text{recipient})\,]}{\mathbb{E}[\,y(\text{donor}) - y(\text{recipient})\,]}$$
Here $y$ is either the predicted PHQ-8 score or the fraction of paralinguistic claims that match the donor's measured features, and expectations run over swap pairs. $T=1$ means the output fully follows the transplanted evidence. $T=0$ means it ignores it. Alongside $T$ we report three rates. The false-claim rate is the share of paralinguistic claims contradicted by measurement. The interviewer-only confabulation rate is the share of rationales that still assert patient-specific symptoms when no patient turns exist. The persistence rate is the share of original-session claims repeated verbatim after a swap that should have reversed them.
Baselines
The naive reference point is a prior-only predictor. It outputs the training-set mean PHQ-8 and the most frequent rationale claims in the training data, regardless of input. Its $T$ is zero by construction. Any audited system whose rationale overlaps with this stereotype as much as with its own measured features is, on our definition, recycling priors. This is the psychiatric analogue of the ungrounded baseline in the HDI paper.
The strongest existing system is MLlm-DR, which reports CCC 0.72 on E-DAIC and 73% expert agreement with its rationales (MLlm-DR). No reported result was found on Lacuna for any measure of MLlm-DR rationale faithfulness to input features. If released weights are unavailable, we reimplement its two-stage recipe on the same E-DAIC split and first confirm that we get within a small margin of 0.72 CCC before auditing, so we audit a faithful copy rather than a weakened one. For the transcript pipeline, PDIMC reports F1 0.90 on DAIC-WOZ (PDIMC). We rerun its theme-summarization stage with the same Qwen2.5 model and audit the summaries. The shortcut reference is the interviewer-only E-GCN at F1 0.88 (DAIC-WOZ prompt bias). Any LLM whose score follows the interviewer-only variant as closely as the full session cannot claim to be reading the patient.
The competitor that most directly threatens the case for free-text rationales is a feature-to-template system. It trains a gender-normalized SVM on the same acoustic features, in the style of Hewlett Sanchez et al., who report 81.3% speaker-level accuracy on their WCGS subsample, and it emits one fixed sentence for each measured feature past a threshold. Its rationales are faithful by construction, so its $T$ for claims should be near 1. It gets the same data split and a matched hyperparameter search budget.
The ablation that isolates our key ingredient is to rerun the full audit using only the removal conditions, original versus no-acoustics, in the manner of HDI. If removal and swap rank the audited systems the same way, the swap adds nothing and we will say so.
To "beat" these references, an LLM system would need three things at once. Its claim $T$ must be clearly above the prior-only predictor. Its false-claim rate must be comparable to the template system's. And clinicians must rate its rationales more useful than the templates in a blinded comparison.
Why not just…?
Why not just compute HDI on psychiatric data? HDI is included as the removal-only ablation, and it is a cheap, sensible first step. Removing the acoustic stream tells you whether the model uses acoustics at all, but not whether it uses the values. A model that switches to "depressed-sounding" language whenever any feature stream is present would score well on removal and fail the swap. The ablation comparison shows directly whether that case occurs.
Why not have a strong LLM check each rationale against the evidence? CounselBench found LLM judges reached only F1 0.5 at detecting clinical failure modes and gave high factuality scores to responses clinicians flagged (CounselBench). We use an LLM only to split rationales into claims. That step is checked against annotators on a subset, and verification itself rests on measurements and humans.
Why not skip the LLM and pair an acoustic classifier with templated explanations? This is a real threat, and we include it as a baseline rather than arguing it away. Its known weakness is that acoustic features are brittle across speakers (Cummins et al.), and its explanations cannot connect what the patient said to what they sounded like. If the blinded clinician comparison finds templates as useful as LLM rationales, and templates also have fewer false claims, the honest conclusion is that current LLM rationales should not be deployed for this task. That would be a publishable and useful outcome.
Why not ground the model in the DSM-5 with retrieval? Retrieval supplies criteria, not patient evidence. Evidence from other clinical settings suggests extra context can raise error rates (Tang et al.), and grounded support rates can mask prior reliance (HDI). We include an InterMind-style retrieval-augmented configuration as an audited system. Its $T$ answers this question directly.
Decisive Checks
The first check validates the verifier. On the 300-claim listening subset, human agreement with the automatic tercile rule must be substantially above chance before any false-claim rate is reported. If it is not, we fall back to human verification for all paralinguistic claims.
The second check is the main test. We compute score and claim $T$ for MLlm-DR, the PDIMC summarizer, a retrieval-augmented LLM, the template system, and the prior-only predictor on gender-matched swaps. We expect LLM claim $T$ well below the template system's. We expect score $T$ to exceed claim $T$, meaning the numeric head reads the features while the rationale head recycles priors, and we would count that dissociation as the headline finding.
The third check measures the interviewer channel. In the interviewer-only variant, we count how often patient-specific symptoms are still asserted, and we test whether the claims that do flip under swaps are predicted better by interviewer question features than by donor acoustics.
The fourth check is the cross-gender arm. It tests whether apparent tracking is really tracking of gender-linked acoustic shifts, which is the confound FairReFuse documents.
The fifth check is the removal-only versus swap ablation. We report any ranking disagreement between the two, since that disagreement is the evidence that substitution measures something removal cannot.
Two limitations are stated in advance. E-DAIC's test set is small, so all $T$ estimates carry bootstrap intervals over swap pairs. And feature-level swaps may produce combinations that no real speaker would produce. We bound that effect by restricting donors to the same interview question and, as a robustness arm, to within-corpus nearest neighbors in speaker embedding space.
References
- The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness
- MLlm-DR: Towards Explainable Depression Recognition with MultiModal Large Language Models
- Predicting Depression in Screening Interviews from Interactive Multi-Theme Collaboration
- InterMind: Doctor-Patient-Family Interactive Depression Assessment Empowered by Large Language Models
- DAIC-WOZ: On the Validity of Using the Therapist’s prompts in Automatic Depression Detection from Clinical Interviews
- CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmark of Large Language Models in Mental Health Counseling
- Using Prosodic and Spectral Features in Detecting Depression in Elderly Males
- An investigation of depressed speech detection: features and normalization
- FairReFuse: Referee-Guided Fusion for Multi-Modal Causal Fairness in Depression Detection
- Evaluating large language models on medical evidence summarization