Word-Aligned Prosody for Small Dementia-Screening Language Models

Speech-based dementia screening has a puzzle at its center. Models that read the transcript of what a patient said beat models that listen to how it was said, often by more than twenty points of accuracy. Yet the transcripts used in practice come from automatic speech recognition (ASR), which tends to delete the hesitations, fillers and false starts that carry the clinical signal. Most attempts to add the audio back either compress a whole recording into one vector of summary statistics, or turn pauses into a handful of text symbols. Both throw away the information about which word was spoken slowly, flatly, or after a long silence.

This note proposes a different route. For each word in the transcript, we compute a small vector of continuous prosodic measurements: the pitch (fundamental frequency, $F_0$) level, range and slope across the word, its energy, its duration normalized by the number of phones, the silence before and after it, and voice-quality perturbations such as jitter (cycle-to-cycle pitch wobble) and shimmer (cycle-to-cycle loudness wobble). A small learned projection turns that vector into an embedding, which is added to that word's input embedding in a compact text model already specialized on clinical picture-description transcripts. The addition passes through a gate that starts at zero, so the system begins exactly as the text-only model and moves away from it only if prosody helps. Only the projection, the gate, and the model's normalization parameters are trained. The central claim is testable: aligning prosody to individual words carries diagnostic information that the transcript and whole-recording acoustic statistics do not, and this advantage should grow as the transcript gets noisier.

Core Claim and What Is New

The claim has two parts, and each can be falsified on its own. The first concerns granularity. Prosody tied to specific words should outperform the same prosodic measurements pooled over the utterance. This mirrors the vision-language finding of A Visual Attention Grounding Neural Model for Multimodal Machine Translation, where grounding source words to image regions beat appending a global image vector, which decoders learned to ignore. MAF reports the same pattern for phrase grounding. The dossier notes that this word-level grounding has not been carried over to continuous acoustic prosody. The second part concerns robustness. Prosody's contribution should increase as ASR word error rate rises, because prosody is measured from the waveform and does not depend on recognition quality. This addresses the tension between the text-dominance results (Learning Language and Acoustic Models for Identifying Alzheimer's Dementia From Speech) and the noisy-ASR argument summarized in Multimodal Speech Fusion for Cognitive Health Assessment.

The closest existing work is Infusing Acoustic Pause Context into Text-Based Dementia Assessment. It inserts four duration-bucket pause tokens and a disfluency token into Whisper transcripts, and in a separate variant lets a frozen BERT cross-attend to wav2vec 2.0 frames at pauses. Our proposal differs in three ways. It attaches continuous measurements to every word rather than discrete symbols at pauses. It covers pitch, energy and voice quality, not only timing. And it adapts the language model through its normalization layers rather than adding a heavy cross-attention block on a frozen trunk. Alzheimer's Disease Detection Using Co-Attention Mechanism for Acoustic and ASR-Transcribed Text Features fuses spectrogram, MFCC (mel-frequency cepstral coefficients, a standard spectral summary), wav2vec 2.0 and DistilBERT features through co-attention. It has no word-level alignment, and it trains many more parameters on a small dataset. We treat it as the strongest existing system to beat.

Research Program

Each direction plays a different role here. Prosodic feature extraction supplies the signal. Domain specialization supplies the backbone and the data-efficiency argument. The cross-modal representation gap literature supplies the main design constraint: the prosody embedding is a foreign distribution entering a text model, and it must not disturb that model's internal activations.

Stage one specializes each modality separately. This follows the finding of Leaner and Faster that small models trained on cross-modal alignment from scratch are unstable, and that mastering each modality first helps. On the text side, we take a BERT-base-scale encoder, the same size used by Braun et al. so comparisons stay fair. We continue its masked-language-model pretraining on DementiaBank Pitt picture-description transcripts, following the corpus-specialization recipe that let small models compete in Enhancing medical coding efficiency through domain-specific fine-tuned large language models and VeriGen. ADReSS is drawn from Pitt, so every evaluation speaker will be excluded from this pretraining corpus, and we will report the overlap check. On the prosody side, we take word timestamps from Whisper forced alignment and compute per-word descriptors with standard pitch and voice-quality extraction. Each speaker's descriptors are normalized by that speaker's own median. This is a partial defense against the microphone, dialect and age confounds the dossier flags as the main weakness of acoustic features.

Stage two connects the two sides with few trainable parameters. The input to layer one for word $i$ becomes

$$h_i = e_i + g \cdot W p_i,$$

where $e_i$ is the word's token embedding (shared across its sub-word pieces), $p_i$ is its prosody descriptor vector, $W$ is a small linear projection, and $g$ is a scalar gate initialized at zero. Besides $W$ and $g$, we train only the LayerNorm parameters inside the attention blocks, plus the classification or regression head. This follows Tuning LayerNorm in Attention, which treats injecting a new modality as a distribution-shift problem and finds that re-centering and re-scaling normalization layers beats LoRA while training a tiny fraction of weights. That result was shown only for vision into large language models, and its transfer to acoustic inputs in a small encoder is part of what we test. The zero-initialized gate is our answer to the worry, raised in Gap 3 of the dossier, that a weak modality pollutes a strong one: at initialization the system is exactly the specialized text model.

We also measure the representation gap directly. For each layer, we record the cosine distance between hidden states with and without the prosody input, and plot it against whether that input helped. The aim is a diagnostic that tells us where prosody enters the computation, not a decoration. Generating free-text diagnostic rationales, and the autoregressive drift Understanding and Bridging the Modality Gap for Speech Translation documents for long generations, are deliberately out of scope. The available benchmarks have no rationale annotations, and adding generation would confound the central question.

Baselines

The naive reference point is the ADReSS challenge's own baselines from Alzheimer's Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge. Acoustic ComParE features with linear discriminant analysis reach 0.625 test accuracy. Linguistic features reach 0.75 test accuracy and 5.20 RMSE on Mini-Mental State Examination (MMSE) score prediction. These numbers tell us whether anything is working at all.

The strongest classical system on ADReSS is the grid-searched logistic regression of Tackling the ADReSS Challenge. It reports 77.8% test accuracy for classification and a best test RMSE of 4.4388 for regression. Its gap between 92.7% cross-validation and 77.8% test accuracy is the overfitting warning our low-parameter design responds to. We will report its published numbers alongside ours and not re-tune it.

The strongest neural system on ADReSSo is the co-attention model of Shao and Fang, which reports 83.15% test accuracy against the official ADReSSo baseline of 77.46%. We will rerun it with its published architecture on our identical Whisper transcripts, with the same hyperparameter search budget in GPU-hours as our model, and report both the rerun and the published figure.

The baseline that matters most for the claim is our own specialized text-only encoder, with gate fixed at zero. It is fine-tuned with the same LayerNorm-plus-head parameter budget, and separately with full fine-tuning, so that "prosody helps" cannot be confused with "a better text model helps". A second baseline is the Braun et al. recipe: discrete pause and disfluency tokens inserted into the same transcripts and fed to the same specialized encoder. For it, the file only provides results on a German cohort (AUC 71.4% for healthy versus mild cognitive impairment, 80.5% for mild impairment versus Alzheimer's, 89.6% for healthy versus Alzheimer's). We found no reported result on Lacuna for pause-token insertion on ADReSS or ADReSSo, so we will run it ourselves.

Three ablations isolate the key ingredient. In the pooled ablation, the same descriptors are averaged over the recording and added as a single extra token. In the shuffled ablation, descriptor vectors are randomly permuted across words within each recording, which keeps every summary statistic but destroys alignment. In the timing-only ablation, only the pause and duration descriptors are kept. Beating the text-only and pause-token baselines shows prosody adds information. Beating the pooled and shuffled ablations shows that word alignment is what adds it.

Test sets are tiny. One subject is about 2.1 accuracy points on ADReSS's 48-subject test set and about 1.4 on ADReSSo's 71. We will therefore report means over at least ten seeds, subject-level bootstrap confidence intervals, and leave-one-subject-out cross-validation on the training sets. We will not claim a win whose interval overlaps the competitor's. Concretely, beating the strongest baseline means a seed-averaged ADReSSo accuracy above the rerun co-attention model with non-overlapping bootstrap intervals, or a statistically matched accuracy at an order of magnitude fewer trained parameters together with a clear win under degraded ASR.

Why not just…?

Why not just fine-tune a domain-specialized text model on the transcripts and stop? This is the most serious threat. The text-dominance literature predicts it will match us on clean manual transcripts, and we expect it may. Our claim is narrower: the gap should open as transcripts degrade. ADReSS ships both manual CHAT transcripts and audio, so we can run the same speakers through Whisper models of increasing size and plot each method's accuracy against word error rate. If the prosody model's gain over text-only stays flat or near zero across that curve, the robustness half of our claim is wrong, and we will say so.

Why not just insert pause tokens, as Braun et al. do? Pause tokens capture timing, which may be most of the useful signal. The timing-only ablation tests this directly. If timing-only matches the full descriptor set, the honest conclusion is that pitch and voice quality add nothing for Alzheimer's screening at this sample size. That outcome would fit Yancheva et al.'s view in Using linguistic features longitudinally to predict clinical scores that the disease is mainly cognitive-linguistic rather than motor. It would still leave a contribution: continuous timing values per word versus four buckets, which the comparison with the pause-token baseline measures.

Why not just append utterance-level acoustic statistics (openSMILE-style functionals) to a text classifier? Late fusion of summary statistics is cheap and resists overfitting, and Tackling the ADReSS Challenge shows simple linear models are hard to beat on these cohorts. The pooled ablation is exactly this alternative inside our model, and the shuffled ablation is its stricter cousin. If either matches the aligned model, the granularity claim fails and the proposal reduces to a robustness study.

Why not just train a bigger end-to-end fusion of wav2vec 2.0 and a text encoder? Shao and Fang's co-attention model is that approach and is the number to beat. With 166 ADReSSo training subjects, we expect its many trainable parameters to be fragile across seeds, and the seed-variance comparison will show whether that is so. If it beats us with tight intervals, the low-parameter route has no advantage. The fair conclusion would then be that richer learned acoustic features beat hand-designed per-word ones. As a follow-up, we would replace $p_i$ with wav2vec 2.0 features pooled within each word.

Proposed Experiments and Decisive Checks

The main experiments use ADReSS, for binary classification and MMSE regression, and ADReSSo, for classification from audio and ASR only. Metrics are accuracy, F1 and AUC for classification and RMSE for MMSE. The German multi-center cohort used by Braun et al. would allow a direct comparison on healthy versus mild cognitive impairment, but it is included only if access can be arranged. The HDS-R Japanese cohort of Kato et al. is a possible test of language transfer, where the only reference point is the adjusted $R^2 = 0.50$ from prosody alone.

Three checks decide the proposal. The first is the shuffle test: aligned prosody must beat within-recording shuffled prosody with non-overlapping intervals, or word alignment is not doing the work. The second is the ASR-degradation curve on ADReSS: the prosody model's margin over the specialized text-only model should rise as word error rate rises from manual transcripts to the smallest Whisper model. The third is a do-no-harm check: on manual transcripts, the gated model should never fall meaningfully below text-only, and the learned gate value together with the per-layer hidden-state distance should show prosody entering where it helps rather than shifting activations uniformly. Passing all three would show that word-level prosody is a useful and cheap addition to small clinical language models. Failing the first while passing the second would show that prosody matters only as a backup for bad transcripts, which is still a result clinicians deploying ASR need to know.