Borrowed-Label Speech Markers for Dementia Detection When Cases Are Rare

Speech carries early signs of Alzheimer's disease. Patients pause longer, speak more slowly, and fill their sentences with "uh", "um", and restarts. The current best audio-only detectors, such as a fine-tuned HuBERT, a large speech model pretrained on unlabeled audio, reach about 78% accuracy on a small benchmark with equal numbers of patients and controls. They cannot say why they flag a patient, and nobody has tested them when patients are a small minority, which is the normal situation in a memory clinic. We propose a detector whose only diagnosis-trained part is a small additive model over about ten named speech measurements that clinicians already use. Those measurements come either from signal processing or from detectors trained on labels borrowed from a different disorder, stuttering, so no dementia label is spent on them. A black-box channel from the pretrained speech model is allowed into the prediction only if it earns its place under a heavier penalty.

The claim is that as positive cases become scarce, this design loses less accuracy than fine-tuning the large model. The reason is that almost all of its learned capacity is fixed without diagnosis labels, so the rare class only has to fit a handful of shape functions. The design also yields explanations that are faithful by construction, and it gives a measurable answer to a question the field has not asked: how much diagnostic signal in modern speech models lies outside the markers clinicians already know?

Core Claim

The obvious combination of the four directions would be "fine-tune HuBERT, add class reweighting, add post-hoc saliency." It inherits the known weaknesses of each part. We instead separate the problem into three stages. First, concepts are estimated label-free, or with labels from another task. Second, a very low-capacity model maps those concepts to diagnosis. Third, a residual channel from the pretrained model is admitted reluctantly.

We predict three things. At balanced prevalence the concept model should come close to black-box fine-tuning. At 10–25% prevalence it should beat it on area under the precision–recall curve. The residual channel should be selected rarely when labels are few, and when it is selected, its contribution should be small and inspectable.

Relation to Prior Work

The closest work is Exploring multi-task learning and data augmentation in dementia detection with self-supervised pretrained models. It fine-tunes HuBERT on the ADReSS benchmark and adds stuttering detection on the SEP-28k corpus as an auxiliary task, raising accuracy from 74.2% to 78.3%. We reuse its central insight, that stuttering-type disfluencies are shared with dementia. We change the role of that auxiliary data. They mix it into shared weights as a second training objective. We make the stuttering detectors the explicit intermediate variables the diagnosis must pass through. The same work evaluates only on balanced data and reports that augmentation stops helping beyond five times the data. That plateau constrains us: we do not rely on synthetic audio to fix imbalance.

Self-supervised Learning is More Robust to Dataset Imbalance motivates the imbalance argument. On long-tailed image benchmarks it shows that supervised training discards features not needed for the majority classes, while label-free training keeps them. It also introduces rwSAM, which reweights training toward low-density regions of representation space. Our concept layer is trained with no diagnosis labels at all, so by the same argument it cannot be distorted by diagnostic skew. rwSAM serves as the strongest imbalance-aware version of the black-box competitor. That paper studies images only, and whether its finding transfers to pathological speech is itself part of what we test.

For the concepts, we rely on established clinical findings. SLaCAD uses the ratio of voiced duration to pause duration as an early marker. A Preliminary Study of Speech Prosody-Based Relationship with HDS-R Scores links speech rate and response latency to a cognitive scale. A comparison study on patient-psychologist voice diarization supplies silence ratio and the separation of patient from interviewer. Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models found that transcript models lean heavily on fillers. Its saliency method also over-flagged frequent short words like "okay." That failure constrains us: explanations computed after the fact on unconstrained models can reflect dataset frequency rather than pathology, which is why our explanations are built into the model.

The additive-plus-reluctant structure borrows its method from A reluctant additive model framework for interpretable nonlinear individualized treatment rules. That model fits a sparse linear base and penalizes nonlinear terms more heavily so they must "earn" inclusion. We apply the same logic one level up: the uninterpretable deep channel must earn its way in over the interpretable concepts. The metric-learning dementia model in Robust and interpretable AI-guided marker for early dementia prediction in real-world clinical settings shows that transparent models can survive real clinic cohorts, but it uses cognitive tests and MRI, not voice. Assessing the Interpretability of Machine Learning Models in Early Detection of Alzheimer's Disease provides our alignment metric: Kendall tau rank agreement between model feature importance and expert rankings.

Research Program

Concept layer. The concept layer produces, per recording of the patient's speech, roughly ten named quantities. Signal processing supplies the voiced-to-pause ratio, mean and variability of pause length, speech rate from syllable-nucleus counting, pitch variability (monotone prosody), and response latency after the interviewer stops, with the interviewer located by diarization. A HuBERT model fine-tuned only on SEP-28k clip labels supplies per-minute rates of interjections (fillers), sound and word repetitions, prolongations, and blocks.

No Alzheimer's label touches this layer. Each concept is standardized against control speakers from the pooled corpora. Within-site standardization is also tested, to address the recording-hardware sensitivity documented in ADReSSo.

Diagnosis model. For a speaker with concept vector $c$ and pooled HuBERT embedding $h$, the model is

$$\text{logit}\,p(y=1\mid c,h)=\beta_0+\sum_{k=1}^{K} f_k(c_k)+w^\top r,\qquad r=h-\hat{A}c,$$

Here $f_k$ is a low-degree spline shape function for concept $k$. $\hat{A}$ is a linear map fit by least squares on unlabeled speakers, so $r$ is the part of the embedding the concepts cannot explain. $w$ is a sparse weight vector penalized $\lambda$ times more strongly than the spline coefficients. $\lambda$ is chosen by nested cross-validation on training speakers. When $w=0$ the model is fully auditable: each $f_k$ can be plotted, and a prediction decomposes exactly into per-concept contributions.

Class imbalance is handled only through prior correction of the intercept and a prevalence-matched decision threshold. We add no resampling, so any robustness we observe comes from the architecture.

Explanation checks. We run three. First, intervention faithfulness: replace one concept with the control-median value and measure the change in prediction. Additive structure makes this exact, but we verify that it matches re-extracting the concept from audio edited to shorten pauses. Second, rank agreement (Kendall tau) between concept importance and a ranking elicited from clinicians. Third, correlation of each $f_k(c_k)$ with the Mini-Mental State Examination (MMSE) score.

Baselines

The naive reference points are the majority-class predictor and the ADReSS challenge acoustic baseline. That baseline, linear discriminant analysis on the ComParE feature set, reports 0.625 test accuracy in the ADReSS Challenge. Under imbalance the majority-class predictor's accuracy rises while its recall is zero, which is why we report precision–recall area and sensitivity at fixed specificity rather than accuracy alone.

The strongest existing method is HuBERT fine-tuned jointly on dementia diagnosis and SEP-28k stuttering. It reports 78.3% test accuracy on ADReSS, with 74.2% for single-task fine-tuning Exploring multi-task learning. It is the right competitor because it uses exactly the same unlabeled pretraining and the same borrowed stuttering labels as our concept layer. The only difference is whether those labels shape a black box or define named intermediates. We will first re-run it and confirm we are close to 78.3% on the balanced split. We then run it at every prevalence level with class-weighted loss, with focal loss, and with rwSAM from Self-supervised Learning is More Robust to Dataset Imbalance. All variants get the same speakers, the same GPU-hour budget, and the same number of hyperparameter trials as our model.

We also include a transparent hand-crafted baseline: logistic regression and random forest on the 88 eGeMAPS acoustic features. As a reported reference, the fused acoustic-plus-ASR system in ADReSSo reaches 78.87% on that separate dataset.

Two ablations isolate our key ingredients. The first removes the SEP-28k detectors and keeps only signal-processing concepts, which tests whether borrowed labels matter. The second sets $\lambda=1$, letting the residual enter unpenalized, which tests whether reluctance matters.

Beating the strongest baseline means two things. At balanced prevalence we should be within a few points of 78.3% accuracy. At 25% and 10% prevalence we should show a higher mean precision–recall area than every HuBERT variant, with the gap exceeding the spread across resampled training draws.

Why not just…?

Why not just fine-tune HuBERT with class weights or rwSAM and explain it with SHAP? This is the main threat on accuracy, and our imbalance sweep settles it directly. On explanation, the evidence is against it. Saliency on transcript models flagged frequent harmless words Detecting Linguistic Characteristics, and SHAP rankings swing with architecture, down to negative agreement with experts Assessing Interpretability. We will compute SHAP on the fine-tuned HuBERT, aggregated over time windows that line up with our concepts, and compare its expert rank agreement and intervention faithfulness against ours. If the black box wins on precision–recall area at low prevalence, we will report it plainly. The design would then retreat to its role as an auditable companion model.

Why not just use eGeMAPS or ComParE with a linear model? These features are transparent in form but mostly not clinically named. They overfit badly on small cohorts: 92.7% cross-validation accuracy fell to 77.8% on test in Tackling the ADReSS Challenge, though that system also used transcripts, and the audio-only acoustic baseline in the ADReSS Challenge scored 0.625 on test. Ten fixed concepts give far fewer degrees of freedom than 88 generic features. The eGeMAPS baseline at matched prevalence shows whether this matters. If eGeMAPS matches us, the contribution shrinks to naming the features, and we would say so.

Why not just compute pauses and speech rate and skip the learned detectors? This is a real possibility, since a pause ratio alone may carry most of the signal. The signal-processing-only ablation tests it. The learned disfluency concepts justify their cost only if they add precision–recall area at low prevalence. A plausible reason they might is that fillers and repetitions are exactly what Chen et al.'s stuttering transfer appeared to exploit.

Why not just transcribe with ASR and prompt a language model? Transcription error is about 60% on this speech ADReSSo. In addition, text discards latency and prosody. We include a Whisper-plus-prompting run in the style of Alzheimer's disease detection based on large language model prompt engineering as a secondary reference only, because it spends no diagnosis labels at all.

Proposed Experiments

The main experiment uses ADReSS: 108 training and 48 test speakers, balanced. We keep all 54 control training speakers and subsample Alzheimer's training speakers to 54, 18, and 6, giving prevalences of 50%, 25%, and 10%. Each level uses 30 random draws, and every method sees identical draws. On the test set we report AUROC, precision–recall area recomputed at the matching test prevalence by speaker reweighting, and sensitivity at 80% specificity. We report bootstrap intervals because a 48-speaker test set is small.

The DementiaBank Pitt corpus is naturally skewed; its majority baseline is 79.8% in Detecting Linguistic Characteristics. It serves as a second, non-synthetic imbalance test, with speaker-disjoint splits that exclude ADReSS speakers. For MMSE regression on ADReSS we compare against the reported test RMSE of 6.14 for the acoustic baseline and 5.20 for the linguistic baseline the ADReSS Challenge. Cross-site transfer from English training to German speakers in PARLO tests whether language-light concepts transfer better than HuBERT fine-tuning. No reported result was found on Lacuna for PARLO, so this is exploratory.

Three checks decide the proposal. The first is whether the gap in precision–recall area between our model and the best HuBERT variant grows in our favor as prevalence falls. If it does not, the imbalance claim fails. The second is how often, and by how much, the residual channel is selected across draws. This is our estimate of diagnostic signal outside known markers, and we expect it to be selected more often as labels increase. The third is whether edited-audio interventions reproduce the model's per-concept contributions, and whether clinician rank agreement exceeds that of SHAP on the black box. If the concept model loses on both robustness and faithfulness, the idea is wrong, and that negative result is itself worth publishing given how much the field has invested in fine-tuning large speech models on balanced benchmarks.