Teaching Speech Models to Hear Word-Finding Failure, Not Just Disfluency

Speech-based dementia detectors are usually judged by one number: classification accuracy on a small benchmark. That number cannot tell a model that hears lexical retrieval failure (the person knows the concept but cannot find the word, so they pause, substitute a generic word, or circle around it) apart from a model that hears something merely correlated with diagnosis, such as general hesitancy, voice quality, or the speaker's sex. Medical imaging ran into the same problem with generated brain MRIs. Image-quality scores rewarded sharp-looking images that were anatomically wrong, and the fix was a metric defined over clinically meaningful quantities. We propose to carry that lesson into speech, in two parts. The first is a training method: an audio model learns, alongside the diagnosis, to predict word-level retrieval markers that are computed automatically from time-aligned transcripts. Examples are the pause before a content word, how rare the produced word is, and how far it sits semantically from the previous word. The second is an evaluation: a fidelity test that asks whether the model's internal representation separates patients from controls more than it separates men from women, and whether its score responds specifically to pauses placed where word-finding happens.

The claim fits in two sentences. The best current audio model gains accuracy by also learning to detect stuttering. Stuttering is a motor-fluency disorder, not a word-access disorder, so that gain may come from generic disfluency rather than from retrieval failure. Replacing stuttering with supervised retrieval markers should match its accuracy, and should beat it on fidelity tests that accuracy cannot see, while still needing only audio at test time.

What the literature supplies and where it stops

On the clinical side, several studies show that retrieval failure in dementia is visible at the level of individual words and transitions between words. GPT-D damages a GPT-2 model by masking half of its attention value weights. It then classifies transcripts by the ratio of perplexities between the intact and damaged models, and finds that the damaged model reproduces the high-frequency vocabulary and low lexical diversity of Alzheimer's speech. Lindsay et al. split a verbal fluency task into 10-second bins and show that people with mild cognitive impairment shift from semantic toward phonemic search. In that study word count alone sat at chance (AUC 0.50), while search-strategy features reached 0.76. Toro-Hernández et al. find that more concrete words and less semantic variability track network hypoconnectivity in Parkinson's disease. These papers motivate our auxiliary targets: they show that word-level retrieval properties carry diagnostic signal that coarse counts miss. All of them work on text, however, and so need transcripts at test time. The ADReSS challenge itself (Luz et al.) relied on manual CHAT transcripts.

On the multi-task side, Chen et al. is the closest prior work and the one we must beat. They fine-tune HuBERT on ADReSS audio jointly with stuttering detection on SEP-28k, which raises accuracy from 74.2% (single task) to 78.3%. Their stated rationale is that hesitations and repetitions in dementia resemble stuttering events phonetically. That rationale is also the weakness. Stuttering labels teach the backbone where speech is disfluent, not whether a disfluency sits at a word-access point. Their evaluation is accuracy alone. They also tried speaker gender as an auxiliary task, which for a clinical model is a nuisance variable rather than a helpful signal.

On the fidelity side, WASABI evaluates generated MRIs by segmenting them into regional volumes and computing a multivariate Wasserstein distance (an optimal-transport distance between two distributions) against real cohorts. Its decisive sanity check is a ranking: on real ADNI scans, only WASABI ranked the healthy-versus-Alzheimer's difference above the male-versus-female difference, while FID tracked scanner style. AG-LDM goes further and puts a frozen segmentation network into training as an extra loss, reducing subcortical volume errors by 15–20%. Our proposal borrows from both. From AG-LDM we take the idea that clinically meaningful structure should be supervised directly rather than hoped for. From WASABI we take the ranking test, applied to a speech model's representation instead of to images. Neither paper touches language.

Core Claim

Supervising an audio backbone with time-aligned lexical retrieval markers yields a dementia model whose evidence is specific to word-finding. Three measurements should show this. The model should be at least as accurate as stuttering-based multi-task learning on ADReSS. It should show a larger ratio of disease separation to sex separation in its representation. Its score should rise more when silence is spliced before low-frequency content words than when the same amount of silence is spliced at random positions. Stuttering supervision is predicted to help accuracy but to fail the third test, because it teaches sensitivity to disfluency regardless of position.

Research Program

Retrieval markers. For every training recording we force-align the transcript to the audio. On ADReSS and DementiaBank Pitt the transcripts are manual; we also run an off-the-shelf speech recognizer to test robustness. From the alignment we compute frame-level and word-level targets. The first is the silent and filled pause duration immediately preceding each content word. The second is the log corpus frequency of each content word. The third is the fastText cosine distance between consecutive content words, following the search measures of Lindsay et al. and the schizophrenia trajectory work (Trajectories through semantic spaces). The fourth is a binary flag for generic substitutions ("thing", "stuff") and circumlocution, as an indicator of word-finding attempts. We also compute a contrast target: pause duration before function words. This lets the model learn that hesitation before a content word differs from hesitation anywhere.

Model. We use HuBERT, as in Chen et al. On top of it sit three heads: an utterance-level head for AD-versus-control classification, a regression head for MMSE score, and lightweight frame-level heads that predict the retrieval markers at aligned positions. Transcripts are used only to build these targets. At inference the model consumes audio alone, which resolves the tension the file identifies: linguistic features beat acoustic ones on ADReSS, but transcription is the bottleneck.

Optimization. The diagnosis loss is defined per utterance, while the marker losses are defined per frame and outnumber it by orders of magnitude. This is exactly the gradient-magnitude imbalance that RotoGrad addresses, using magnitude homogenization plus learned task-specific rotations of a feature subspace. We treat RotoGrad as an ablation, not as the contribution. The default is a fixed loss weighting tuned on validation folds, and we test whether gradient homogenization adds anything beyond it.

Fidelity evaluation. Let $z$ be the mean-pooled backbone representation, projected onto its top $k$ principal components fit on training speakers. We use small $k$ (5–10), because the cohorts are small. We fit a Gaussian to each group and compute the closed-form 2-Wasserstein distance $W_2$. The representation fidelity ratio is

$$F = \frac{W_2(\mu_{\mathrm{AD}}, \mu_{\mathrm{HC}})}{W_2(\mu_{\mathrm{male}}, \mu_{\mathrm{female}})},$$

where $\mu_g$ is the fitted distribution of $z$ for group $g$. A value of $F$ above 1 reproduces the WASABI sanity ranking. We repeat this with estimated signal-to-noise quartiles as a second nuisance axis. The counterfactual test is

$$\Delta = \mathbb{E}\left[s(x \oplus p_{\text{content}})\right] - \mathbb{E}\left[s(x \oplus p_{\text{random}})\right].$$

Here $s$ is the model's AD probability for a control speaker's recording $x$, and $\oplus$ splices a fixed 800 ms silence before low-frequency content words or at matched random word boundaries. A positive $\Delta$ means the model treats pauses at retrieval points as evidence; $\Delta \approx 0$ means it reacts to pausing in general. Finally, we correlate model scores with MMSE, the criterion GPT-D used ($r$ up to −0.64).

Baselines

The naive reference points are the ADReSS challenge baselines. Acoustic features with LDA reached 0.625 test accuracy and 6.14 MMSE RMSE; linguistic features reached 0.75 accuracy and 5.20 RMSE (Luz et al.). They show how much any learned model adds and that text carries more signal than audio.

The strongest audio competitor is MD-MTL from Chen et al., which reports 78.3% accuracy on the ADReSS test set with HuBERT plus SEP-28k stuttering detection; their single-task HuBERT reported 74.2%. It is the strongest competitor because it is the same backbone and the same multi-task idea with a different auxiliary signal, so it isolates exactly our question. We will re-run it ourselves on the same splits, with the same augmentation recipe (their best was 5x scale), the same number of tuning trials, and the same seeds. Comparisons will use our reproduction, with the reported 78.3% as a sanity anchor. Beating it means matching its accuracy within the seed-level confidence interval while showing a significantly larger $F$ and $\Delta$. An accuracy gain alone, on a test set this small, would not count.

The strongest overall system uses text: GPT-D reports AUC 0.89 and 85% accuracy on ADReSS and AUC 0.77 zero-shot on CCC. We do not expect an audio-only model to beat it on ADReSS. We include it to measure the remaining gap, and we will compute $\Delta$ for it too, by inserting pause tokens into its input transcripts, to see whether a text model is retrieval-specific.

Three ablations isolate the key ingredient. In the first, the marker heads are trained on pause duration at all word boundaries without the content/function distinction; this tests whether position matters or whether "more pause supervision" suffices. In the second, stuttering and retrieval markers are trained jointly. In the third, the content-word markers are replaced with targets computed on shuffled alignments, as a placebo with identical loss magnitude.

Why not just…?

Why not just add MMSE regression as the auxiliary task? It is cheap and clinically grounded, and we include it in every variant. But MMSE is a single utterance-level number per speaker. It gives the backbone no information about where in the signal the impairment lives, so it can be fit by any correlate of severity, including voice ageing. The MMSE-only multi-task model is one of our arms. If it matches the marker model on $F$ and $\Delta$, our mechanism is unnecessary and we will report that.

Why not just extract retrieval markers with a speech recognizer at test time and feed them to a classifier? This is the most serious threat. Lindsay et al. show that hand-built search features reach AUC 0.76, and the ADReSS linguistic baseline beats acoustic features. We will run exactly this pipeline: recognizer, then forced alignment, then the same markers, then gradient boosting, the same classifier family Toro-Hernández et al. used. Its weakness is that recognizers degrade on hesitant elderly speech, as Chen et al. note. Our model needs the recognizer only at training, where errors average out over many words. The experiment settles this directly. If the feature pipeline matches our model on accuracy and MMSE correlation even under recognizer transcripts, the contribution reduces to "transcript-free inference," and we will state it that way.

Why not just use GPT-D on recognizer transcripts? GPT-D is strong and needs no training. It still inherits transcript errors, and it does not see timing at all: a pause before "the... um... faucet" is invisible once transcribed as "the faucet". We will report GPT-D on recognizer transcripts against manual ones. If the drop is small, a text route is viable and our audio model's advantage is limited to the timing-based $\Delta$ test.

Why not just keep stuttering supervision, which already works? It does raise accuracy. Our prediction is that it raises it for the wrong reason, and $\Delta$ measures exactly that. If the stuttering model shows a $\Delta$ as large as ours, the hypothesis that it encodes generic disfluency is refuted.

Proposed Experiments

Training uses ADReSS plus DementiaBank Pitt recordings for marker supervision. Any Pitt speakers who also appear in ADReSS are removed, so the test set does not leak. SEP-28k is used for the stuttering arms. We evaluate classification and MMSE on the ADReSS test set, and zero-shot generalization on CCC conversations, where GPT-D's 0.77 AUC is the reference. Because ADReSS is small, every number is reported over at least ten seeds with bootstrap confidence intervals. $F$ is tested against a label-permutation null, and $\Delta$ uses paired tests across control speakers.

The decisive check is the comparison between our marker model and the reproduced stuttering model on $\Delta$. The proposal succeeds if retrieval supervision produces a positive $\Delta$ significantly larger than stuttering supervision and the all-boundaries ablation, with accuracy no worse than stuttering supervision. A secondary check is whether $F$ exceeds 1 for our model and not for single-task HuBERT. That would carry WASABI's "disease above sex" criterion into speech. A third check is whether the advantage persists when marker targets come from recognizer transcripts rather than manual ones; this determines whether the method scales beyond hand-annotated corpora. As an exploratory transfer test, we apply the frozen backbone with a new linear probe to the French Dem@Care fluency recordings. Lindsay et al. report AUC 0.76 there with hand-built features. Representations that truly encode word-access failure should transfer across tasks and languages better than ones tuned to picture-description disfluency.