Self-supervised speech models are networks pretrained on large amounts of unlabeled audio. When dementia is detected from voice recordings, these models beat hand-engineered acoustic features inside a single benchmark, and then lose to those same features when tested on a different cohort. On one English benchmark, embeddings from wav2vec 2.0 reach 78.9% accuracy (Alzheimer Disease Recognition Using Speech-Based Embeddings From Pre-Trained Models). When a model is trained on English and tested on Greek, a support vector machine on classical openSMILE paralinguistic features (pitch, jitter, shimmer, energy and pause statistics) scores F1 0.72, against 0.64 for multilingual XLSR-53 embeddings (Cross-Lingual Alzheimer's Disease Detection Based on Paralinguistic and Pre-Trained Features). Hand-built timing features also survive English-to-Spanish transfer (AUC 0.75) better than lexical ones (0.64) (Automated Speech Markers of Alzheimer Dementia: Test of Cross-Linguistic Generalizability). With fewer than 200 speakers, a classifier on a 768- or 1024-dimensional embedding has room to latch onto whatever separates the classes in one recording setup: microphone, room, session order, or language. The descriptor classifier cannot do this, because its input space is small and every axis is physically meaningful.
We propose to keep the foundation model and restrict what the classifier may look at. Using unlabeled speech only, we learn which directions of a frozen speech model's hidden states linearly encode classical paralinguistic descriptors: pitch contour, voice quality, loudness, speech rate and pause structure. The diagnostic classifier is then trained only on the projection onto that "descriptor-anchored" subspace. A single rank parameter moves the method from "descriptors only" to "the full embedding". Our central prediction is that in-cohort accuracy rises with rank while cross-cohort accuracy peaks at low rank. If that holds, the low-rank projection gives a principled operating point that beats both the raw descriptors and the full embedding when the test cohort is new.
Core Claim
The paralinguistic information that transfers across cohorts is already present in speech foundation models. It is drowned out in small-sample training by high-variance directions that encode corpus identity. Anchoring the classifier to the descriptor-predictive subspace should recover it. This subspace is estimated without diagnosis labels, which is where the label efficiency comes from. The specific new claim is that this subspace, at matched rank, transfers better across cohorts than three alternatives: the raw descriptors, the full embedding, and subspaces of equal size chosen by variance (PCA), at random, or by phonetic content.
Research Program
Let $h_t \in \mathbb{R}^D$ be the frozen hidden state of a speech model at layer $l$ and frame $t$. Let $d_t \in \mathbb{R}^m$ be a vector of frame-aligned classical descriptors computed with openSMILE over a short window, where $m \approx 25$ for the eGeMAPS low-level descriptors. We fit a ridge regression $d_t \approx A h_t$ on unlabeled speech. We then take the top-$k$ right singular vectors of $A$, extended beyond rank $m$ by reduced-rank canonical correlation between $h$ and a lagged descriptor window, to form an orthonormal basis $U_k \in \mathbb{R}^{D \times k}$. The classifier sees only
$$z = \mathrm{pool}_t\,\big(U_k^\top h_t\big),$$
where pooling uses mean, standard deviation and a small learned attention over 5-second segments. The 5-second segment length follows the finding in Alzheimer Disease Recognition Using Speech-Based Embeddings From Pre-Trained Models that this length balances context against sensitivity to short disfluencies. Speaker-level predictions average over segments.
The regression is fit on speech that carries no diagnosis labels. The main condition uses conversational and spontaneous speech from outside the evaluation cohorts. A transductive condition also uses the unlabeled target-cohort audio, and is reported separately because it can absorb target-channel statistics. Read-speech corpora are a known poor source of conversational cues (Towards Joint Modeling of Dialogue Response and Speech Synthesis based on Large Language Model), so we compare anchoring on read speech against spontaneous speech.
We sweep four backbones. XLSR-53 is required for the Greek transfer. We also use wav2vec 2.0, TRILLsson, which was distilled specifically for paralinguistics (TRILLsson: Distilled Universal Paralinguistic Speech Representations), and emotion2vec (emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation). We sweep every layer, because the Perceiver study on a frozen Universal Speech Model found intermediate layers (layer 11 of 32) best for pathological speech while deep layers discard non-phonetic variation (Detecting Speech Abnormalities with a Perceiver-based Sequence Classifier that Leverages a Universal Speech Model). That work used a single cohort and a fixed layer. We ask a different question: which directions within a layer transfer.
This method sits between two lines of prior work. The motivating papers (Chen et al., García et al.) show that descriptors transfer but stop at SVMs and decision trees. The Unified Pathological Speech prompt-tuning work also constrains the design. It discretizes HuBERT features into cluster units (Unified Pathological Speech Analysis with Prompt Tuning), which throws away continuous prosody. This is why we project continuous features rather than tokenizing them.
Baselines
The naive reference point is a majority-class predictor and the official challenge baselines. For ADReSSo the acoustic baseline is 64.5% accuracy and the late-fusion baseline is 78.87% (Detecting Cognitive Decline Using Speech Only: The ADReSSo Challenge). For ADReSS the best acoustic baseline is 0.625 test accuracy (Alzheimer's Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge).
The strongest cross-cohort competitor is IS10-Paralinguistics features with an SVM from Chen et al.. It reports F1 0.72 for English-to-Greek classification on ADReSS-M and RMSE 4.788 for Mini-Mental State Examination (MMSE, a standard cognitive test score) regression. We rerun it with our cross-validation protocol and our hyperparameter budget so the numbers are directly comparable. Beating it means a higher Greek-test F1 with a bootstrapped 95% interval on the difference that excludes zero, or, if intervals overlap, matching it while also exceeding it on the in-cohort English split.
The strongest in-cohort competitor is full-embedding wav2vec 2.0 with 1D convolutions and segment averaging, as in Gauder et al.. It reports 78.9% ADReSSo test accuracy. No reported cross-cohort result was found on Lacuna for that exact pipeline. Its XLSR analogue in Chen et al. reports F1 0.6441 English-to-Greek.
We also run Compositional Feature Alignment. It trains orthogonal task and domain heads on a frozen encoder and then fine-tunes the encoder into that structure (Enhancing Compositional Generalization via Compositional Feature Alignment). We adapt it to speech by using recording session or corpus as the domain label. Every method gets the same frozen backbone, the same pooling head, and the same Optuna budget of 50 trials. Model selection uses source-only cross-validation, with the target cohort touched only at final test.
The key ablations hold rank $k$ fixed and replace $U_k$ with three alternatives: top-$k$ PCA directions, a random orthonormal $k$-subspace, and a "phonetic" subspace fit to predict phone posteriors. These isolate whether descriptor alignment, rather than simply having fewer dimensions, drives transfer.
Why not just…?
Why not just use the openSMILE features with an SVM, since they already transfer best? This is the most serious threat. Chen et al. show it winning cross-lingually. However, raw acoustic features overfit badly inside a cohort: the ADReSSo organizers report strong cross-validation but a test-set collapse (ADReSSo), and ADReSS acoustic features trail transcripts (ADReSS). Descriptors also carry frame-wise estimation noise that foundation-model features may smooth over. If our method only ties the SVM across cohorts and loses in-cohort, the honest conclusion is that foundation models add nothing transferable here. The rank sweep is designed to report exactly that.
Why not just concatenate descriptors with the full embedding? Early fusion of eGeMAPS, TRILL, Allosaurus and wav2vec 2.0 gave no in-cohort gain over wav2vec 2.0 alone (Gauder et al.). Concatenation still leaves the classifier free to use the corpus-specific directions. We include it as a baseline and expect it to track the full embedding across cohorts.
Why not just regularize harder or reduce dimension with PCA? Generic capacity control could plausibly explain any gain we find. The matched-rank PCA and random-subspace ablations exist to test this. If PCA at rank $k$ matches descriptor anchoring, the core claim fails.
Why not just remove domain directions, as in CFA or adversarial training? Using a single English source cohort provides almost no domain variety. The file also flags that channel effects such as low volume and damped spectra overlap with pathology, so removing domain directions can erase disease markers. Anchoring defines the useful subspace positively rather than by subtraction, and the CFA baseline tests whether this distinction matters.
Proposed Experiments
The primary test is ADReSS-M English-to-Greek classification and MMSE regression, with no Greek labels used. The secondary test is ADReSS or ADReSSo English within-cohort. Because these two sets share Pitt-corpus origins, we do not treat transfer between them as cross-cohort. Where access allows, we add the Chilean Spanish cohort from García et al. (AUC 0.75 timing-only reference) and the Carolinas Conversation Collection audio. The latter moves from picture description to conversation, a shift that the text-only prompt method of Farzana and Parde handles for transcripts but never for acoustics.
We report F1, AUC and RMSE with speaker-level bootstrap intervals, because small-cohort variance is large (Gauder et al.). Four checks decide the outcome. First, the cross-cohort F1 curve over rank must peak at low $k$ while the in-cohort curve keeps rising. Second, the descriptor-anchored subspace must beat PCA and random subspaces of matched rank. Third, it must beat the IS10 SVM, at 0.72 F1, on Greek. Fourth, a label-budget sweep from 10% to 100% of English training speakers must show the anchored probe degrading more slowly than the full-embedding probe. A final diagnostic trains a probe to predict corpus identity from $z$ versus from the full embedding. It checks whether anchoring really removes cohort-identifying information, which would explain any transfer gain rather than simply coexisting with it.
References
- Alzheimer Disease Recognition Using Speech-Based Embeddings From Pre-Trained Models
- Cross-Lingual Alzheimer’s Disease Detection Based on Paralinguistic and Pre-Trained Features
- Automated Speech Markers of Alzheimer Dementia: Test of Cross-Linguistic Generalizability
- Towards Joint Modeling of Dialogue Response and Speech Synthesis based on Large Language Model
- TRILLsson: Distilled Universal Paralinguistic Speech Representations
- emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation
- Detecting Speech Abnormalities With a Perceiver-Based Sequence Classifier that Leverages a Universal Speech Model
- Unified Pathological Speech Analysis with Prompt Tuning
- Detecting Cognitive Decline Using Speech Only: The ADReSSo Challenge
- Alzheimer’s Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge
- Enhancing Compositional Generalization via Compositional Feature Alignment
- Domain Adaptation via Prompt Learning for Alzheimer's Detection