Teaching Dementia Classifiers to Cite Their Own Pauses

Speech-based dementia screening works well inside one clinic and breaks when moved to another. The best current systems insert "pause tokens" into speech transcripts, so a language model sees where and how long a speaker went silent, not just what they said. These models reach high scores within a corpus. Across recording sites they drop sharply, because they learn the site: the interviewer's prompting style, the tablet microphone, the ambient noise. We propose to train the classifier to write a short, checkable justification alongside every diagnosis. The justification must point to specific pauses and hesitations in the patient's own speech. It is not produced by a teacher language model. A deterministic program computes it from the audio timestamps, so every claim in it can be verified against the recording. The hypothesis is that forcing the model to predict this patient-side timing evidence steers its internal representation away from site shortcuts, and that this shows up as a smaller performance drop when the model is tested on an unseen clinic or corpus.

The idea borrows the logic of explanation-augmented classification, in which a model generates a rationale together with its label so that predictions rest on real evidence rather than dataset artifacts. It applies that logic to the clinical version of the "cross-platform gap" known from social media, where the recording site plays the role of the platform. The claim is narrow and falsifiable. A timing-grounded rationale objective should reduce cross-site degradation more than three alternatives do: label-only fine-tuning of the same model, simply feeding the same timing features as inputs, and training the model to be blind to site identity.

Where the Evidence Points

Two groups of papers motivate the proposal. The first shows that timing matters and transfers. Infusing Acoustic Pause Context into Text-Based Dementia Assessment bins pause durations into tokens P1–P4, adds a disfluency token, and runs German BERT with cross-attention over Wav2Vec 2.0 audio embeddings. It reports AUC 89.6% for healthy versus Alzheimer's and finds that pause-aware text beats audio alone. Optimizing Pause Context does the same in Cantonese with grid-searched pause thresholds and shows that the best thresholds differ across languages and disease stages. Automated Speech Markers of Alzheimer Dementia finds that timing features transfer zero-shot from English to Spanish (AUC 0.75) while lexical features do not (0.64). Fusing the two drops transfer to 0.65. Both deep-model papers use direct classification with no explanation, and neither tests cross-site transfer.

The second group shows that site shortcuts are the real failure. Classifying Dementia in the Presence of Depression finds that transcript models learned interviewer prompting behavior. Its cross-corpus BERT result falls to 56.0% unweighted average recall (UAR, the mean of per-class recall), and 68% of misclassified dementia patients had comorbid depression. Motor, Cognitive, or Corpus? probes nine self-supervised audio encoders and reports a 12.5 balanced-accuracy drop under recording-condition shift, along with a failure to separate Parkinson's disease from dementia.

The method comes from rationale-augmented work outside clinical speech. Beyond Classification trains an audio language model to co-generate emotion labels and explanations that quote the transcript, and reports large gains over label-only training. It ignores timing entirely, and it notes that teacher-generated rationales carry the teacher's biases. S³-MSD uses bootstrapped rationales and preference optimization to reduce hashtag shortcuts in sarcasm detection, and transfers from Twitter to Reddit. CATCH instead removes platform information architecturally, using a variational autoencoder, and beats GPT-4 across platforms.

We differ from all three in one specific way. Our rationales are computed from measurements of the patient's own speech rather than generated by a teacher model. This removes the hallucination and teacher-bias problems, and it lets us test directly whether rationale content, and not just an extra loss term, drives robustness.

Core Claim

Let $x$ be a pause-tokenized transcript of the patient's speech, with interviewer turns removed by diarization. Let $y$ be the diagnosis and $r$ a structured rationale. The rationale is a short text built by a rule program from word timestamps. For example: "four pauses over two seconds in the first thirty seconds; long pause before the noun 'Hütte'; filler precedes three of five content words; speech rate 1.4 words/s". Each clause references a position in $x$.

The model is trained on

$$\mathcal{L} = -\log p_\theta(y \mid x) - \lambda \sum_t \log p_\theta(r_t \mid x, r_{<t}),$$

where $\theta$ are the model parameters, $r_t$ is the $t$-th rationale token, and $\lambda$ weights the rationale term.

At test time the model receives only $x$. It must infer the evidence itself from the pause tokens; it is never handed the computed features.

The claim is that including $r$ narrows the gap between within-site and cross-site UAR more than any baseline below. It should also do so because of what $r$ says, not merely because of the extra loss term.

Research Program

The backbone is a mid-sized German-capable decoder language model with low-rank adapters. This is the same model family as the Mistral-3.1 baseline in The PARLO Dementia Corpus, which introduces a nine-clinic, 208-subject German dataset with manual pause and filler transcription and CSF/PET-confirmed diagnoses.

We add the pause and disfluency tokens to the vocabulary. The pause bin boundaries are tuned per task on training folds only, following the lesson from Optimizing Pause Context.

Domain pre-training is a supporting ingredient, not the headline. Before supervised training, we continue pre-training on all available unlabeled pause-tokenized transcripts from training sites. This teaches the model what the new tokens mean. It is motivated by specialized-pretraining results such as MonoCoder, where a compact model trained only on its target domain beat much larger general models. We treat pre-training as an ablation because the corpora are small. We never include audio from the held-out site, which avoids transductive leakage.

The rationale generator is a transparent rule program. We will release it, together with a validation study in which two clinicians rate whether each rationale clause is clinically meaningful. Rationale templates contain only patient-side quantities. Interviewer behavior, recording-device information and absolute loudness are excluded by design.

Baselines

The naive reference point is chance UAR (50% for binary tasks) together with a majority-class predictor. The zero-shot reference is prompting Mistral-3.1 on the same transcripts. The PARLO Dementia Corpus reports UAR 45.6% on picture description alone and 66.0% with recall transcripts added. We rerun it with pause tokens described in the prompt.

The strongest existing method is the pause-token BERT with Wav2Vec 2.0 cross-attention from Infusing Acoustic Pause Context. It reports AUC 71.4% for healthy versus MCI on verbal fluency, 80.5% for MCI versus Alzheimer's on picture description, and 89.6% for healthy versus Alzheimer's. These results are within-cohort only. For cross-corpus transfer, the reference is the 56.0% UAR German BERT result from Classifying Dementia in the Presence of Depression.

We reimplement the Braun et al. model on our splits, with the same diarized transcripts and the same hyperparameter search budget: 20 trials per model, tuned on inner folds of the training sites only. It counts as beaten only if our model has higher cross-site UAR with non-overlapping bootstrap 95% intervals, while staying within two AUC points of it within-site.

The timing-only classical model re-implements the feature set of Automated Speech Markers of Alzheimer Dementia as a sparse classifier, including its 0.75 AUC cross-language setup where feasible. It is the most dangerous baseline, because it is simple and already robust.

Four ablations isolate the key ingredient, each using the same backbone, data and tuning budget:

  • Label-only. Set $\lambda = 0$.
  • Shuffled rationales. Pair each rationale with a different subject of the same class. This keeps label information but breaks the link to the input.
  • Shortcut rationales. Describe interviewer turn counts and recording properties instead of patient timing. If rationale content matters, this should hurt transfer.
  • Free-form rationales. Generate them with a teacher language model, to test whether measurement grounding matters.

Why not just…?

Why not just prompt a frontier language model? The file's evidence argues against it. Zero-shot Mistral-3.1 reaches only 45.6% UAR on PARLO picture descriptions (The PARLO Dementia Corpus), well below supervised pause-aware models. CATCH reports that a small disentangled model beat GPT-4 across platforms. We still run the zero-shot condition, so the comparison is on our own splits rather than taken from the literature.

Why not just feed the computed timing features into a classifier? This is the real threat, since our rationales are built from those same features. Automated Speech Markers of Alzheimer Dementia shows timing-only features transfer at 0.75 AUC. It also shows that naive fusion with lexical content degrades transfer to 0.65. Our bet is that using timing as a training target lets the model keep lexical information without that degradation. If the timing-only classifier matches us cross-site, the honest conclusion is that the language model adds nothing under shift, and we will report that. We also run a feature-as-input variant of our own model to separate "target" from "input".

Why not just make the model blind to site with adversarial training on clinic labels? PARLO's nine clinics make this cheap, and it follows the disentanglement spirit of CATCH. We include a gradient-reversal variant with the same backbone and budget, plus a combination of both objectives. Site-blindness removes information that predicts the site, but it does not say which information should drive the diagnosis. Site and diagnosis prevalence are partly confounded across clinics, so blinding may also remove signal. The comparison settles this empirically.

Why not just use an audio-only encoder? Motor, Cognitive, or Corpus? shows that such encoders lose 12.5 balanced-accuracy points under condition shift. Infusing Acoustic Pause Context found pause-aware text beats audio alone. We therefore include audio-only models only as a secondary reference.

Proposed Experiments and Decisive Checks

We evaluate three shift settings:

  • Leave-one-clinic-out on PARLO.
  • Train on NSC, test on the multi-center tablet cohort, and the reverse. The file suggests the tablet cohort overlaps PARLO, so overlapping subjects will be deduplicated.
  • An exploratory Cantonese run on CU-Marvel. It is within-corpus only, and checks that the objective does not hurt the setting where Optimizing Pause Context reports 0.924 accuracy and 0.832 F1 for healthy versus major neurocognitive disorder.

Tasks follow Braun et al.: onset (healthy versus MCI), monitoring (MCI versus dementia) and exclusion (healthy versus dementia). Metrics are UAR and AUC, with subject-level bootstrap intervals over five seeds. The cohorts are small, so we pre-register the primary comparison: cross-site UAR against the label-only ablation.

We run four decisive checks:

  1. Gap reduction. Rationale training must shrink the within-site to cross-site gap relative to label-only, and must match or beat timing-only features cross-site.
  2. Content matters. Shuffled and shortcut rationales must not reproduce the gain. Shortcut rationales should enlarge the gap.
  3. Faithfulness. Editing long pauses to short ones in test inputs should shift both the predicted label and the generated rationale. We also measure rationale claim accuracy against ground-truth timestamps, and report the counterfactual edits separately.
  4. Confounding. On NSC subjects with depression annotations, error rates in the depressed subgroup should fall relative to label-only. This tests whether patient-timing supervision helps separate cognitive from affective slowing, a failure documented in Classifying Dementia in the Presence of Depression.

Continued pre-training is reported as an additional ablation. We make no claim that depends on it.