Speech-based screening for dementia rests on measurable signs: patients pause more and longer, say "uh" more often, use fewer distinct words, convey less of a picture's content, and build simpler sentences. These signs are easy to compute from a careful human transcript. In deployment, though, the transcript comes from automatic speech recognition (ASR). On impaired speech ASR misrecognizes words at error rates near 60% (The ADReSSo Challenge), which corrupts exactly the signs clinicians trust. The field has responded in two ways, both unsatisfying. Black-box audio-text networks raise accuracy but give no clinically checkable reasons. Hand-crafted feature pipelines stay readable but degrade on noisy input. Both are also trained to output a yes/no diagnosis, even though cognitive decline is graded and the people who most need assessment sit near the boundary.
We propose to train a neural network that reads raw audio and estimates a fixed set of named clinical speech measures, such as pause rate, filled-pause rate, lexical diversity, picture-content units, and subordination. Its targets are the values those measures take on the manual transcript. Every prediction must then pass through those measures to a small additive model that outputs a continuous cognitive score and a per-measure deviation from healthy speakers. The supervision costs nothing extra, because corpora with manual transcripts already contain the ground-truth measures. The test is narrow and falsifiable. On audio-only data, do the network's estimates of these measures beat computing them directly from the ASR transcript? And does that translate into better graded prediction of cognition without giving up readability?
Core Claim
A concept bottleneck model, meaning a network forced to predict human-named intermediate quantities and to make its final output only from those quantities (Concept Bottleneck Models), is useful for speech in a specific way. Its value is not merely that the output can be read. Its value is that it can recover clinically defined measures that the obvious pipeline (ASR, then a feature script) destroys. If the learned estimates of pause, lexical, and syntactic measures correlate better with their manual-transcript values than script-computed ASR features do, then an additive model on top can match strong black-box fusion on continuous cognitive scoring. It would also stay inspectable and correctable measure by measure. If the learned estimates do not beat the script, the bottleneck has no reason to exist, and the honest recommendation becomes a transparent model on ASR features. The experiments are built so that this outcome would show clearly.
How This Relates to Prior Work
The motivating evidence comes from multimodal speech work that achieves accuracy through unconstrained fusion. Chatzianastasis et al. fuse an architecture-searched spectrogram network with BERT through tensor decomposition (Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia), and they name explainability as future work. A co-attention model weights wav2vec2 frames using spectrogram, MFCC, and ASR-text embeddings (Alzheimer's Disease Detection Using Co-Attention Mechanism). CogniVoice combines Whisper, BERT, and DisVoice phonation, articulation, and prosody features in a product-of-experts ensemble that discourages single-modality shortcuts (CogniVoice). All three predict from opaque internal representations, and all except CogniVoice target only a binary label. We keep their encoders but replace their decision pathway with named measures.
Prior interpretability work on these corpora is post hoc. Karlekar et al. show through saliency maps on transcript models that networks attend to fillers and sentence-initial "and" (Detecting Linguistic Characteristics of Alzheimer's Dementia). The same analysis also shows saliency over-flagging frequent but non-pathological tokens. We take their findings as a source of candidate concepts, not as a method. Two papers supply method pieces. The first is the clinical-knowledge alignment loss of Pang et al., which penalizes a bottleneck model when perturbation-measured concept importance departs from expert importance tiers; it raised out-of-domain F1 on RaabinWBC from 43.64% to 49.75% (Integrating Clinical Knowledge into Concept Bottleneck Models). The second is the non-negativity and effective-concept sparsity used against leakage in Multimodal Concept Bottleneck Models. Both were developed only for images. Our concepts differ from theirs in kind. Ours are computable quantities with free ground truth on manual transcripts, so no new expert annotation is needed for concept labels.
The dimensional framing comes from work arguing that clinical states lie on continua. Population normative modeling of brain MRI found deviations that scale continuously with symptom severity rather than clustering by diagnosis (Breaking the Norm). Sparse canonical correlation found cross-diagnostic symptom axes in youth connectivity (Linked dimensions of psychopathology). We borrow the normative idea, which is to express each person as deviations from a healthy reference. Our scope is narrower than "transdiagnostic." The available speech corpora cover only the Alzheimer's/MCI spectrum, so we test dimensional modeling within cognitive decline: graded MMSE, the MCI boundary, and two-year decline. Cross-disorder claims are left for when suitable speech data exist. For readability we adopt the expert-agreement metric of Haddada et al., Kendall rank correlation between model feature importance and clinician rankings (Assessing the Interpretability of Machine Learning Models in Early Detection of Alzheimer's Disease). That work found agreement ranging from perfect for trees to negative for k-nearest neighbors on tabular data.
Research Program
Concept inventory and labels. We fix roughly fifteen measures across three groups. The temporal group covers pause rate, mean silent-pause duration, filled-pause rate, speech rate, and articulation rate. The lexical-semantic group covers type-token ratio over a fixed window, Cookie Theft content-unit coverage from a keyword list, and pronoun-to-noun ratio. The syntactic group covers mean utterance length, subordination ratio, and the rate of sentence-initial coordinating conjunctions. Phonation and prosody measures from DisVoice, such as F0 variability, jitter, and shimmer, are computed directly from audio, as in CogniVoice. Ground truth for the transcript-dependent measures comes from manual CHAT transcripts in ADReSS and the Pitt corpus, using forced alignment for timings. This is the free supervision.
Concept estimator. A Whisper-style audio encoder, plus an ASR-text encoder over the noisy transcript, feeds one regression head per measure. Each head is trained to predict the manual-transcript value. Because the audio carries pause and hesitation evidence that ASR drops, we expect the largest gains on temporal and filler measures. Lexical and content measures should show smaller gains, and this prediction can be checked.
Normative layer. Each estimated measure is converted to a deviation from the training-set cognitively normal speakers, adjusted for age and sex. A speaker is thus represented as a profile of signed deviations. The reference group is small, on the order of 80 controls in ADReSS, so we report deviations with bootstrap intervals and do not claim a population norm.
Additive head. The deviation profile feeds an additive model that outputs MMSE. In the spirit of RAITR, each measure is linear by default and receives a penalized smooth term only if cross-validation supports it. Binary labels, when needed, are a threshold on the score, not a separate head. The head sees nothing but the measures. We train the bottleneck sequentially, estimator first and head second, to limit leakage of unnamed information. As an ablation, we add Pang-style alignment to expert importance tiers. These tiers would be elicited once from speech-language pathologists and also used for the Kendall evaluation.
Corrigibility. Because each measure is named, a clinician can override one estimate, for example after hearing that a long pause was a cough, and see the score change. We measure how much replacing predicted measures with manual-transcript values improves error. This is the standard bottleneck intervention test, and it tells us how much remaining error lives in concept estimation versus the head.
Baselines
The naive reference is a predictor of the training-set mean MMSE, together with majority-class accuracy for the binary view. On ADReSSo the published challenge baseline is the floor for credibility: audio-only MMSE regression at RMSE 5.28, late-fusion classification at 78.87% accuracy, and two-year decline prediction at 68.75% accuracy and 66.67% mean F1 (The ADReSSo Challenge). On ADReSS, with manual transcripts available, the linguistic baseline MMSE RMSE is 5.20 (The ADReSS Challenge).
The most important baseline, because it is the most direct threat, is the script pipeline. It computes the same fifteen measures directly from the ASR transcript and audio and feeds them to the same additive head, with identical deviation normalization and identical tuning. It differs from our model in exactly one ingredient: whether the measures are estimated by a supervised network or computed by rule from noisy ASR. Beating it means two things. First, a higher mean correlation between estimated and manual-transcript measure values. Second, a lower MMSE RMSE on held-out audio, with a bootstrap interval that excludes zero difference.
The strongest black-box competitors are the co-attention model at 83.15% ADReSSo accuracy (Co-Attention Mechanism) and, for graded scoring, CogniVoice, which reports MMSE RMSE 2.34 and MCI F1 84.1 on TAUKADIAL (CogniVoice). The 92.08% figure for DARTS+BERT+BLOCK (Neural Architecture Search) is on ADReSS, where manual transcripts are available, so it is not comparable to an audio-only setting. We cite it only as an upper reference. To make comparisons fair, we will retrain an unconstrained version of our own encoder (the same Whisper and text encoders, with a free MLP head and no bottleneck) under the same splits, seeds, and hyperparameter-search budget. That isolates the cost of the bottleneck, the "interpretability tax." We count the proposal a success if this tax is within one MMSE point of RMSE while concept fidelity and expert agreement improve. Exceeding the black box is not required.
Why not just…?
Why not just compute the features from the ASR transcript and fit a linear or tree model? This is the real threat, and the proposal is designed so that it can win. The file's evidence against it is weak. The 0.625 ADReSS accuracy is for acoustic features alone, while the linguistic decision tree reached 0.768 in leave-one-subject-out validation (The ADReSS Challenge). A well-chosen feature set could do considerably better. The script-pipeline baseline above is exactly this alternative with matched features and head. If its measure correlations and RMSE match ours, we will report that the supervised estimator is unnecessary and recommend the script.
Why not just use Whisper plus BERT and explain it afterward with SHAP or saliency? Post-hoc attributions on these corpora have flagged frequent but uninformative tokens (Detecting Linguistic Characteristics). In imaging, unconstrained models learned shortcuts that failed under domain shift (Integrating Clinical Knowledge into CBMs). Neither result proves the same happens in speech. We therefore test it directly. We run SHAP on the unconstrained twin, map attributions onto the fifteen measures through correlation, and compare Kendall agreement with clinician rankings and cross-corpus robustness on TAUKADIAL against the bottleneck. If post-hoc explanations agree as well and the black box transfers better, the bottleneck loses on these grounds.
Why not just regress MMSE directly? Our model does regress MMSE, so the question is what the concepts add beyond a direct regressor. The unconstrained twin is that direct regressor. The bottleneck has to justify itself through three things a scalar regressor cannot offer: per-measure deviation profiles, intervention gains, and a test of whether those profiles separate MCI from normal in TAUKADIAL better than the scalar score does.
Decisive Experiments
All models are trained on ADReSS training data, plus Pitt sessions after removing any speaker who appears in the ADReSSo test set. Wherever speaker IDs cannot be matched, we drop the overlapping material rather than risk leakage. Evaluation is on the ADReSSo audio-only test set, using Whisper-generated ASR.
The first check is concept fidelity. For each measure we compare the Pearson correlation with its manual-transcript value for the learned estimate and for the script computation, using held-out ADReSS speakers whose manual transcripts exist. We predict clear gains on temporal and filler measures and small ones on syntax. The second check is downstream. We report MMSE RMSE and thresholded accuracy on ADReSSo against the challenge baselines, the script pipeline, and the unconstrained twin, plus decline prediction on the ADReSSo longitudinal cohort using change in the deviation profile. The third check is the intervention test: how much RMSE drops when estimated measures are replaced with manual values. The fourth is transfer. We freeze the estimator and refit only the head on TAUKADIAL English and Chinese, comparing MCI F1 and MMSE RMSE with CogniVoice's reported 84.1 and 2.34. Lexical concepts may not carry across languages, and we will report per-group transfer. The fifth check is expert agreement: Kendall rank correlation between the head's measure importances and clinician rankings, with and without the alignment loss.
The proposal stands or falls on the first check combined with the script-pipeline comparison in the second. A supervised estimator that does not recover clinical measures better than ASR plus a script is not worth its complexity.
References
- Detecting Cognitive Decline Using Speech Only: The ADReSSo Challenge
- Concept Bottleneck Models
- Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia
- Alzheimer's Disease Detection Using Co-Attention Mechanism for Acoustic and ASR-Transcribed Text Features
- CogniVoice: Multimodal and Multilingual Fusion Networks for Mild Cognitive Impairment Assessment from Spontaneous Speech
- Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models
- Integrating Clinical Knowledge into Concept Bottleneck Models
- Multimodal Concept Bottleneck Models
- Breaking the Norm: Population-Scale Normative Modeling of Brain Structure in Depression and Anxiety
- Linked dimensions of psychopathology and connectivity in functional brain networks
- Assessing the Interpretability of Machine Learning Models in Early Detection of Alzheimer's Disease
- A reluctant additive model framework for interpretable nonlinear individualized treatment rules
- Alzheimer’s Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge