Scoring Depression Questionnaire Items from What Could Fill the Slot

Models that detect depression from text often work well on the dataset they were trained on and fail on the next one. One reason is that the features they learn are tied to how a particular dataset was collected. This proposal tries a different representation. For each emotionally loaded word a person writes or says, we hide the word and ask a masked language model (a model trained to guess hidden words from their surroundings) which words could take its place. We then compare that list of possible replacements with the replacement list produced by the same hiding procedure applied to the wording of each item of a standard depression questionnaire, the nine-item Patient Health Questionnaire (PHQ-9). The output is nine interpretable item scores per person, all expressed in the model's fixed vocabulary rather than in a feature space learned from one dataset.

The bet is that two kinds of anchoring travel across datasets better than learned dense features. One is a vocabulary that does not change between corpora. The other is a clinical instrument whose items mean the same thing everywhere. We test that bet directly by training on one depression corpus and measuring how much performance survives on others. In two sentences: describe a person's language by the replacement words a language model proposes for its key words, and score that description against the replacement words of each PHQ-9 item. Then check whether this instrument-anchored, vocabulary-anchored representation loses less accuracy across datasets than fine-tuned encoders and questionnaire-similarity baselines.

Core Claim

Representing a word occurrence by its in-context substitutes gives a reference frame that stays fixed across corpora. Substitution-based Semantic Change Detection using Contextual Embeddings showed this for comparing language across historical eras without aligning vector spaces. Its substitute-distribution divergence reached a weighted Spearman correlation of 0.514 on SemEval 2020 Task 1 and GEMS, against 0.452 for static vectors and 0.422 for aligned contextual vectors. We claim the same property helps with a different kind of shift: moving between depression datasets collected on different platforms and under different protocols. That shift is currently handled either by learned feature mapping or not at all.

The new element is the anchor. Generic substitute vectors are matched against substitute fingerprints of the PHQ-9 item texts. The resulting nine scores therefore carry the instrument's construct definitions, not the label quirks of whichever corpus the classifier saw. A second, less obvious benefit follows from masking the target word. The substitute list reflects what the context implies, not the word the speaker actually chose. Putting Words in Context found that language-model layers build a representation of the contextual slot that predicts human-given substitutes better than the target word's identity does (0.41 versus 0.29 cosine similarity). This matters because people often hedge or conceal symptoms (Kang et al.). A softened word in a bleak context should still yield bleak substitutes.

Research Program

The representation is built in three steps. First, we pick slots in a person's text: content words with affective or somatic load, plus a sample of other content words so that recall does not hinge on a seed lexicon. For each slot we extract the top substitutes from BERT, following the masking-and-top-k indexing pipeline of Large Scale Substitution-based Word Sense Induction. That pipeline keeps the method cheap enough for interview transcripts and social-media timelines, since it compressed Wikipedia-scale representations from terabytes to about 20GB.

Generative substitutes sometimes come out in the wrong part of speech. To guard against this, we apply the hidden-state agreement filter from GeneSis. That filter keeps a candidate only if the model's internal representation with the candidate inserted stays close to the original's. GeneSis used it to reach 52.1 precision-at-1 on the SemEval-2007 lexical substitution task.

Second, we build an anchor for each PHQ-9 item. We mask each content word in the item text and in a small set of clinician-written paraphrases, then pool the resulting substitute distributions. For example, masking "tired" in "feeling tired or having little energy" yields that item's fingerprint.

Third, we score each person on each item. For a person $p$ and item $i$, the item score is

$$s_{p,i} = \frac{1}{|S_p|}\sum_{t \in S_p} q\big(1 - \mathrm{JSD}(P_t \,\|\, A_i)\big),$$

where $S_p$ is the set of slots in the person's text, $P_t$ is the substitute distribution at slot $t$, $A_i$ is item $i$'s anchor distribution, $\mathrm{JSD}$ is Jensen-Shannon divergence, and $q$ is a quantile transform fit on a neutral reference corpus. The quantile transform is there because raw divergences are distorted by word frequency, a problem documented by Card. A linear model with nine inputs then maps item scores to a total score or a caseness label. We use the standard PHQ-9 caseness threshold, by analogy with the HADS cutoff used in Hatch et al.. Keeping the final model this small is deliberate: it leaves little room to absorb dataset-specific shortcuts.

The analysis side asks two questions. Does the representation shrink the gap between within-dataset and cross-dataset performance? Do the item scores behave like the items, meaning they correlate with item-level self-reports wherever a dataset releases them? Because every score breaks down into concrete substitute words, we can also show which words drive each item on each dataset. That gives a direct test of the "divergency" problem raised by Shen et al., where the same behavior signals opposite mental states on different platforms.

Data and Evaluation Regime

The source corpus is DAIC-WOZ, a set of clinical interviews with a virtual interviewer. Following Burdisso et al., we use participant turns only. That paper showed a classifier trained on the interviewer's prompts alone reaches 0.88 F1 by learning the interviewer's follow-up habits. That number is our warning sign, not a target to beat.

The targets are datasets drawn from the 16 unrestricted-access depression corpora catalogued by Bucur et al.. We prioritize corpora with PHQ-9 totals or item labels, and include at least one social-media corpus and one non-interview format. Where data agreements permit, we add the Twitter and Weibo setting of Shen et al. as a cross-platform stress test, using a multilingual model for the Chinese side.

We report macro-F1 for caseness and Spearman correlation for severity. The headline quantity is the transfer ratio: cross-dataset F1 divided by within-dataset F1, computed with the same model and the same tuning budget.

Baselines

The naive reference point is the majority-class predictor and a bag-of-words logistic regression trained on DAIC-WOZ participant turns. These show how much any method actually adds after transfer.

The strongest general competitor is a fine-tuned BERT classifier trained on the same participant-only source data. It uses the same backbone as our substitute extractor, so differences cannot come from model size. It receives a hyperparameter search matched in GPU hours to the total cost of our extraction plus fitting. It is the method a reviewer would expect to win in-domain, and the open question is how much of that edge survives transfer.

For the cross-platform setting, the strongest existing method in the file is DNN-FATC from Shen et al. It learns non-linear and quantile mappings between platform feature spaces and reported 78.5% F1 on Twitter-to-Weibo transfer, 4.8 points above ARC-t and HFA. We will run our method on that setting if the data can be obtained and compare against that figure. Note that DNN-FATC uses target-platform data to learn its mapping, while our method uses no target data. Matching it would therefore be a meaningful result, not a modest one.

For participant-only DAIC-WOZ in particular, no reported result was found on Lacuna. We will establish the fine-tuned-BERT number ourselves under the protocol above.

Two ablations isolate the key ingredients. The first keeps the PHQ-9 anchoring but replaces substitute distributions with the dense hidden state of the masked slot, and replaces JSD with cosine similarity. This tests whether the fixed vocabulary matters or only the anchoring does. The second keeps the substitute distributions but drops the anchoring, training a regularized classifier over the full substitute vocabulary. This tests whether the instrument matters.

Beating the baselines means a transfer ratio clearly above fine-tuned BERT's, with cross-dataset F1 at least equal to BERT's on most target corpora, and both ablations falling in between.

Why not just…?

Why not just embed the text and the PHQ-9 items with a sentence encoder and use cosine similarity? This is the most serious threat, because it already keeps the instrument anchoring and needs no slot extraction. The first ablation above is essentially this method run at slot granularity. We will also run it at sentence granularity. If it matches our transfer ratio, the fixed-vocabulary claim fails, and we will report the simpler method as the result.

There are reasons to expect a gap. Card's comparison found substitute distributions beat aligned dense vectors when corpora differ, 0.514 versus 0.422. Sentence-level similarity is also dominated by topic and register, which are exactly what changes between interviews and posts. Neither reason settles the question, so the experiment has to.

Why not just prompt a large language model to rate each PHQ-9 item? This is plausible, and it also needs no training. We include it as a zero-shot baseline on every target. It will likely be competitive in raw F1. Its weaknesses are cost on long timelines, sensitivity to prompt wording, and the lack of an inspectable account of which words drove a score. If it dominates on transfer, our contribution narrows to interpretability and cost, and we will say so.

Why not just fine-tune on the source and add domain adaptation? Feature-alignment methods such as Compositional Feature Alignment and DNN-FATC need labeled or unlabeled target-domain data. Bucur et al. show that most depression corpora sit behind agreements, so such data is often unavailable. Our setting assumes no target access. Where target data is available, we report DNN-FATC as an upper reference rather than a direct competitor.

Proposed Experiments and Decisive Checks

The first decisive check is the transfer matrix: train on DAIC-WOZ and evaluate on every target, and, where label formats allow, train on each target and evaluate on the others. Every method above shares the same splits. The claim survives only if substitute-anchored scoring has the highest average transfer ratio and the dense-anchored ablation does not match it.

The second check is construct validity. On targets with item-level labels, each item score $s_{p,i}$ should correlate more strongly with its own item than with the other eight. If the scores show no such item-specific pattern, they are not measuring the items and amount to a general negativity measure.

The third check is a shortcut probe. We reinsert interviewer turns into DAIC-WOZ and measure how much each method's within-dataset score inflates, using the 0.88 interviewer-only F1 from Burdisso et al. as the scale of the possible shortcut. A representation pinned to nine instrument items should inflate least.

The fourth check targets concealment. We take target-corpus posts from people above the caseness threshold, replace explicit symptom words with hedged synonyms, and measure the drop in each method's scores. If masking the target really lets context carry the signal, our scores should fall less than the dense and lexicon baselines.

A negative outcome on the first check, with the dense-anchored ablation equal to the full method, would be informative by itself. It would show that the instrument, not the substitute vocabulary, is what carries depression detection across datasets.