Pseudo-Phonetic Contexts: Transcript-Free Sound-Class Pooling for Small Clinical Speech Cohorts

Many speech disorders do not affect all sounds equally. A voice disorder may show up clearly in nasal consonants and barely at all in voiceless ones. The best small-cohort result in the file depends on this: Automated Voice Pathology Discrimination from Continuous Speech Benefits from Analysis by Phonetic Context cuts read speech into individual phones, trains one small classifier per phone type, and fuses the outputs. On a 60-patient differential diagnosis this reaches 83.37% unweighted average recall, against 75.44% for whole-recording summaries. The method has a practical catch. Cutting speech into phones requires a forced aligner (a tool that matches a known transcript to the audio), and the aligner in turn needs a correct transcript. That rules out spontaneous clinical conversation and severely disordered speech, which are exactly the settings where the approach would matter most.

We propose to replace linguistic phones with sound classes discovered without any transcript. Frames from a frozen self-supervised speech model are clustered into a modest number of acoustic units, and runs of identical units become segments. Speech is then pooled per unit class instead of per phone. The hypothesis is that the gain from phonetic-context analysis comes from grouping acoustically similar sounds so that the disorder's uneven footprint is not averaged away, and does not depend on getting the linguistic labels right. If that holds, the method transfers to spontaneous and unscripted speech. A second claim matters especially for small datasets: the number of sound classes $K$ sets the granularity of the representation. We predict that the best $K$ grows with the number of training speakers, because finer classes give each per-class classifier fewer samples. This turns the unresolved question of how fine-grained features should be into a measurable bias-variance curve.

Core Claim

Grouping frames by discovered sound class recovers most of the advantage of transcript-based phonetic-context decomposition, and it beats uniform pooling of the same frozen features on small clinical cohorts. The grouping must be cheap. It has no trainable parameters beyond small per-class linear models and a sparse fusion layer. On small cohorts this restraint is a requirement, as two results in the file show. In Learning to detect dysarthria from raw speech, adding a single learnable normalization stage dropped performance on TORGO from 82.4% to 74.8% UAR. In On Improving Acoustic Models for TORGO Dysarthric Speech Database, shrinking the model improved results. Both set the design constraint here, which is to add structure rather than capacity.

The scoring rule is written as

$$s(x) = \sum_{k=1}^{K} w_k \, f_k\big(\phi_k(x)\big),$$

where $x$ is a recording and $\phi_k(x)$ is the median and interquartile range of mid-layer foundation-model features over all segments assigned to unit class $k$. The median and interquartile range follow the per-segment summary of the phonetic-context paper. Each $f_k$ is a regularized linear classifier for class $k$. The fusion weights $w_k$ carry an L1 penalty so that only diagnostically useful classes survive. Classes with too few segments in a recording are masked and their weights renormalized.

Relation to Prior Work

Three papers motivate the proposal by documenting how much granularity matters.

  • The phonetic-context paper above supplies the per-context-classifier-plus-fusion recipe that we reuse. It also notes that it cannot run without manual transcripts, and that its gender-imbalanced cohort may have let the classifiers learn sex rather than pathology. We remove the transcript and control for gender explicitly.
  • Alzheimer Disease Recognition Using Speech-Based Embeddings From Pre-Trained Models shows the opposite pattern. Fixed 5-second windows of wav2vec 2.0 features, averaged uniformly, reach 78.9% on ADReSSo, and adding phonetic embeddings from Allosaurus gave nothing. We take this result seriously as the strongest evidence against the proposal. Our version differs in what it groups by: we group frames by sound class, whereas that work fed phonetic information in as extra input features.
  • Detecting Speech Abnormalities with a Perceiver-based Sequence Classifier that Leverages a Universal Speech Model lets learned per-attribute queries (trainable vectors that score which frames to attend to) decide where to look in frozen USM features, reaching 83.1%. It also finds that a middle layer (11 of 32) works best. Our approach fixes where to look by sound identity instead of learning it. The open question is which of the two is safer when the cohort is small. We borrow its mid-layer finding to choose the layer used for clustering.

Two papers constrain the design.

Research Program

Unit discovery. We run k-means on mid-layer features from a frozen WavLM or wav2vec 2.0 model. The features for clustering come from healthy speech plus the training fold only, to prevent leakage between training and test speakers. We sweep $K$ over 8, 16, 32, 64, and 128. Consecutive identical units are merged into segments, and segments shorter than two frames are discarded.

Speaker identity check. Small cohorts let classifiers memorize who is speaking. We therefore measure how well a probe can recover speaker identity and sex from the pooled vectors $\phi_k$. If sex is strongly recoverable, we residualize sex out of the features before classification.

Granularity curve. We subsample training speakers at several sizes. At each size we record which $K$ maximizes held-out UAR. The prediction that the best $K$ grows with $N$ follows from a simple argument: the number of per-class estimates grows with $K$, while the samples available to each class shrink as $1/K$.

Interpretation. We inspect the surviving fusion weights, for example to see whether nasal-like units dominate the voice-disorder task as the transcript-based study found. Where English inversion is reliable, we project unit classes through the inversion model of Speaker-Independent Acoustic-to-Articulatory Speech Inversion to characterize what they correspond to articulatorily.

Baselines

Naive reference. We use the published challenge acoustic baselines. On ADReSS, Alzheimer's Dementia Recognition Through Spontaneous Speech: The ADReSS Challenge reports 0.625 accuracy with ComParE features and linear discriminant analysis. On TORGO, global low-level descriptors reach 65.5% UAR in Learning to detect dysarthria from raw speech. We also run our own frozen-feature mean-pool linear probe as the "do nothing" model.

Strongest uniform-pooling competitor. This is the Gauder et al. pipeline: the same frozen encoder, 5-second windows, a small convolutional scorer, and averaged scores. The reported result is 78.9% accuracy on ADReSSo. We rerun it with the same encoder layer, the same folds, and the same hyperparameter budget, measured in configurations tried. Beating it means a better mean accuracy with a paired bootstrap interval that excludes zero.

Transcript-based upper reference. This is MFA-based phonetic-context pooling (83.37% UAR in the phonetic-context paper). We rebuild it on our encoder features wherever transcripts exist. This is the comparison that tests the core claim. Success means the transcript-free version comes within about two UAR points of it on scripted speech. It does not need to beat it.

Learned-attention competitor. This is a Perceiver-style classifier with per-class queries over the same frozen features, following Soltau et al. (83.1% average accuracy on the Mayo corpus). On TORGO, the strongest reported number is 82.4% UAR with learnable time-domain filterbanks, from Millet and Zeghidour.

Key-ingredient ablation. We keep everything identical but assign segments to $K$ groups at random, preserving the segment-count distribution. If random groups do as well as unit classes, the gain comes from richer pooling statistics, not from sound-class grouping, and the core claim fails. A second ablation computes the median and interquartile range over all frames without grouping.

Why not just…?

Why not just mean-pool foundation-model features and train a linear probe? On ADReSSo this alternative is a real threat. Gauder et al. found that phonetic information added nothing on top of wav2vec 2.0. We expect the proposal's advantage to be largest on voice and motor disorders, which leave localized acoustic traces. On cognitive decline, the signal may be spread across pauses and wording. The experiments report each corpus separately so that a null result on ADReSSo is visible rather than averaged away. A ranking where grouping wins on the voice-disorder corpus and on TORGO but not on ADReSSo would itself be a useful answer to when granularity is needed.

Why not just run a phone recognizer and pool by recognized phones? Recognizers trained on typical speech map disordered sounds onto the phones the speaker intended. On aphasic speech the error rate is 38.63% even after articulatory soft-labeling, per Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling. Discovered units carry no such linguistic prior. We nonetheless include Allosaurus-derived phone pooling as an additional baseline. If it matches our units on dysarthric speech, the simpler story holds and we will report it.

Why not just let attention learn where to look, as the Perceiver does? Attention can in principle discover sound classes on its own. However, its queries are trainable parameters, and the TORGO frontend result shows that even small amounts of added flexibility overfit when there are only about 15 speakers. The speaker-subsampling curve settles this question. Our prediction is that grouping wins at small $N$, while attention catches up or wins as $N$ approaches the size of the Mayo cohort.

Proposed Experiments

Corpora.

  • TORGO, with leave-one-speaker-out evaluation. This is the extreme small-$N$ case.
  • ADReSSo, using the official test split plus speaker-disjoint cross-validation. This is spontaneous speech with no transcripts.
  • The SD/VP corpus, if access can be obtained. It is the only setting where gold transcript-based pooling can be compared directly.

The Mayo corpus appears to be institutional. If it is unavailable, we cite its numbers only as context and do not claim a comparison.

Statistics. All results are speaker-independent and use nested cross-validation for $K$ and regularization strength. We report bootstrap confidence intervals and paired permutation tests, because the file stresses how wide intervals are at these sample sizes.

Decisive checks. The proposal succeeds if four things hold together:

  1. Unit-class pooling beats random-group pooling.
  2. It comes within about two points of MFA-based pooling on scripted speech.
  3. It beats the 5-second-window baseline on at least the voice and motor corpora.
  4. The best $K$ rises with training-set size.

If unit classes do no better than random groups, the idea is wrong. If the sex probe explains most of the gain, the gain is a gender confound and not pathology.