Automatic detectors of dysarthria, a motor speech disorder, report near-perfect accuracy on the two standard corpora. They reach that accuracy even when fed only the silence between words. The reason is that patients and healthy controls were recorded under different conditions, and the detectors learn the room rather than the speech. This proposal builds a detector whose only route to a decision is a small, human-readable table: how acoustically distinct each pair of speech sounds is within a single speaker. Each sound is modeled as a probability distribution over frames from a pretrained speech network. The distinctness of two sounds is a statistical overlap measure between their distributions, not a distance between two average points. A sparse linear model reads the table.
The payoff is twofold. First, the explanation is the model itself: "this speaker was flagged because /i/ and /u/ overlap far more than in controls." That matches how clinicians and phoneticians already describe the disorder, since earlier work found the vowel space shrinks in pathological speech. Second, the model never sees non-speech audio. Recording effects that shift all of a speaker's sounds together tend to cancel when sounds are compared with each other. The open question is whether this interpretable bottleneck keeps enough accuracy once the recording shortcut is removed. The proposal tests that under conditions where black-box models have been shown to cheat.
Core Claim
A relational bottleneck of pairwise divergences between a speaker's own phoneme distributions will do three things on dysarthria detection. It will match or beat black-box classifiers once recording-condition shortcuts are neutralized. It will lose much less accuracy than they do when those shortcuts are removed. And it will produce explanations that track known articulatory degradation. The claim has two ingredients, and each is tested by ablation.
The first ingredient is relational comparison: sounds are compared with each other within a speaker, rather than with a fixed template. The second is distributional comparison: the model uses Bhattacharyya distance, which measures how much two probability densities overlap and is sensitive to spread as well as center. The alternative would be distances between mean vectors.
One property is exact and one is an empirical bet. The exact property is that f-divergences, a family that includes Bhattacharyya distance, are unchanged when the same invertible transformation is applied to both distributions. A linear channel effect in the projected feature space therefore cancels in the table. The bet is how far this carries over to the non-linear effects that additive noise produces in self-supervised features. That is the question the experiments settle.
What Exists and What Is Missing
The divergence-structure idea comes from pronunciation assessment. Integration of multilayer regression analysis with structure-based pronunciation assessment models each phoneme as a Gaussian over hand-crafted spectral features (MFCCs). It builds a matrix of Bhattacharyya distances between phonemes within each speaker and feeds it to layered regression. That system reaches 0.75 correlation with teachers on the English Read by Japanese (ERJ) corpus, against 0.77 agreement between two human experts, and it holds up under simulated vocal-tract warping. We reuse its mechanism, but three things change. The features are learned from a pretrained self-supervised encoder, a network trained on unlabeled audio. The projection is trained end-to-end through the divergence. And the target is clinical detection rather than language teaching.
The clinical motivation comes from two papers. Study of time and frequency variability in pathological speech and error reduction methods for automatic speech recognition found that KL divergence between vowel formant distributions drops by about 18% in pathological Spanish speech. That drop is exactly the kind of signal our table exposes. Its model, however, is an HMM warping system, not a classifier with an explanation. On using the UA-Speech and TORGO databases to validate automatic dysarthric speech classification approaches shows the shortcut directly. Classifiers score 96–100% on silence segments. An MFCC-SVM reaches 81% on speech and 100% on silence. Estimated signal-to-noise ratio (SNR) is 3.7 dB for controls against −7.6 dB for dysarthric speakers. That paper motivates the proposal and also defines its hardest test.
On the interpretable-model side, the closest work treats prototypes as points or near-points. This Looks Like That compares image patches to learned point prototypes. HyperPg puts a one-dimensional Gaussian over the cosine similarity to an anchor direction, reaching 74.3% on CUB-200-2011 versus 61.4% for Euclidean prototypes under the same simplified training. It does not compare full multivariate distributions, and it does not compare prototypes with each other. An Effective Local Prototypical Mapping Network for Speech Emotion Recognition brings vector prototypes to HuBERT features. It reports 77.42% weighted accuracy on IEMOCAP but never checks whether its prototypes encode recording or speaker artifacts.
This Looks Like That... Does it? constrains our design. It shows that similarity in a latent space can track imperceptible artifacts: JPEG compression shifts similarity scores from 5.18 to 1.68. So we do not assume closeness in latent space means anything to a human. We anchor every unit of the bottleneck to a named phoneme contrast and test its meaning against external measurements. Finally, Contrastive Prototypical Network with Wasserstein Confidence Penalty uses Wasserstein distance only as a penalty on output probabilities, not as the matching operation. None of these papers builds a bottleneck out of divergences between a speaker's own sound categories.
Research Program
Features and sound segments. A frozen self-supervised encoder, HuBERT or wav2vec 2.0, produces frame features. A learned linear projection maps them to a low-dimensional space of 16–32 dimensions. The low dimension keeps covariance estimates stable with few frames per phoneme, which addresses the sample-complexity problem noted for 768–1024 dimensional embeddings. Both UA-Speech and TORGO use prompted material, so we align frames to phonemes with the known prompt text. Only frames aligned to phonemes enter the model, which means silence is excluded by construction.
The bottleneck. For each speaker we fit a Gaussian to the projected frames of each phoneme. The covariance is shrunk toward a pooled covariance, because dysarthric speakers may produce few tokens of some sounds. For each selected pair of phonemes $a,b$ with means $\mu_a,\mu_b$ and covariances $\Sigma_a,\Sigma_b$, we compute the Bhattacharyya distance
$$ D_B(a,b)=\tfrac18(\mu_a-\mu_b)^\top\bar\Sigma^{-1}(\mu_a-\mu_b)+\tfrac12\ln\frac{\det\bar\Sigma}{\sqrt{\det\Sigma_a\det\Sigma_b}},\qquad \bar\Sigma=\tfrac12(\Sigma_a+\Sigma_b). $$
The first term compares centers. The second compares spreads, which is where temporal and spectral instability shows up.
Classifier and training. The vector of $D_B$ values, one per selected phoneme pair, goes into an L1-regularized logistic regression. The model predicts dysarthric versus control, and severity class where labels exist. The distance is differentiable, so the projection and the classifier train jointly. The explanation is the handful of nonzero weights, each tied to a named contrast. Because the corpora contain few speakers, all evaluation is leave-one-speaker-out.
Validity checks. We run two. The first replicates the ERJ setup with deep features, to confirm the bottleneck still captures phonetic structure. The second asks whether the learned contrast weights line up with independent acoustic facts: the vowel-contrast collapse reported by Saz et al. on their Spanish corpus, and formant measurements we compute ourselves on UA-Speech vowels.
Baselines
The naive reference point is the MFCC-SVM from Favre et al., rerun under our leave-one-speaker-out protocol. That paper reports 81% accuracy on UA-Speech speech segments and 100% on silence alone. Those two numbers are the reference for "a detector that is mostly reading the room."
The strongest competitor is a black-box classifier on the same features. It fine-tunes the same self-supervised encoder, with attention pooling and a linear head, on the same phoneme-aligned frames. We chose this because Favre et al. found frozen wav2vec 2.0 features partly resistant to the silence shortcut. It is therefore the most accurate plausible system that is not trivially cheating. The file gives no numeric accuracy for it: no reported result found on Lacuna for a self-supervised encoder baseline on UA-Speech or TORGO dysarthria classification. We will establish it ourselves.
The third baseline is a prototype network in the style of LPMN, a point-prototype speech model, run on the same encoder. Its 77.42% weighted accuracy on IEMOCAP shows it is a competitive interpretable speech model, though on a different task.
Two ablations isolate our ingredients. The first keeps the relational table but replaces $D_B$ with Euclidean distance between phoneme means. This removes the distributional part. The second feeds each phoneme's own mean and covariance directly to the classifier, without pairwise comparison. This removes the relational part.
Fairness rules apply to every system. All use the same encoder checkpoint, the same aligned frames, the same speaker folds, and the same hyperparameter search budget of equal trial counts on inner folds. Our model beats the strongest competitor if either of two things holds. It could match the black box within the confidence interval on raw data while losing clearly less accuracy under the shortcut-removal conditions described below. Or it could simply outperform it under those conditions.
Why not just…?
Why not just trim silence with voice activity detection and train the black box? This is the most serious threat, and it is built into our design: every system, the black box included, sees only phoneme-aligned speech frames. The remaining question is whether noise on the speech frames themselves still carries the shortcut. The 11 dB noise-floor difference affects speech as well as pauses. The SNR-equalization test below settles whether trimming is enough. If the black box on trimmed speech holds its accuracy when SNR is equalized, our robustness argument fails. In that case the remaining contribution is interpretability at equal accuracy, and we will report it that way.
Why not just normalize features per speaker (mean and variance normalization) or augment with noise and room simulation? Per-speaker normalization removes shifts and scalings of each feature dimension, which is a subset of what relational divergences cancel. It gives the model no explanation, and it does not touch changes in covariance shape. Augmentation can equalize noise conditions, but it trains the model to ignore noise rather than removing noise as a pathway. Hoffmann et al. also show that robustness training on prototype models can cost heavily in clean accuracy: 75.8% fell to 56.3% on CUB. Both are cheap, so we add them as extra black-box variants inside the same budget. If augmentation plus normalization closes the gap under SNR equalization, we will say so.
Why not just explain the black box after training with saliency or SHAP attributions? Attribution maps on a model that uses recording artifacts can still highlight plausible speech regions. The silence results show that high accuracy is no evidence of legitimate reasoning. We test this directly: we compute Integrated Gradients on the black box in the SNR-equalized setting and check whether its attributions predict the model's accuracy drop. Our model's explanation needs no such check, because the weights are the decision.
Proposed Experiments
The primary benchmarks are UA-Speech and TORGO, each evaluated leave-one-speaker-out, with accuracy and balanced accuracy at the speaker level and 95% confidence intervals from bootstrapping over speakers.
The decisive test is shortcut removal under three conditions. In the first, we add noise to control recordings until their estimated SNR matches the dysarthric group, closing the −7.6 versus 3.7 dB gap. In the second, we train on UA-Speech and test on TORGO, and the reverse. In the third, we pair each utterance with a version passed through a randomly chosen room impulse response and measure how much the prediction changes.
The prediction is specific. The black box and the Euclidean-means ablation should lose substantially more accuracy under SNR equalization than the divergence bottleneck does. The relational ablation should be the most sensitive of all to room-response changes, because it lacks the cancellation property.
For interpretability, we measure three things. The first is the stability of the selected contrasts across speaker folds. The second is the correlation between each speaker's vowel-pair $D_B$ values and independently measured F1/F2 vowel-space area. The third is a deletion test: zeroing a selected contrast should change the prediction by the amount its weight implies. That test holds by construction for the linear readout, so it mainly serves to check the implementation.
For transfer, we reproduce the pronunciation task on ERJ with deep features and compare against the reported 0.75 correlation. We also run the detector on the Saz et al. Spanish corpus to see whether it recovers the vowel-collapse pattern that paper reports.
The idea is refuted if either of two results appears. The divergence bottleneck could lose as much accuracy under SNR equalization as the black box on trimmed speech. Or its selected contrasts could fail to correlate with measured vowel-space shrinkage. Either result would show that the table is not doing what its labels claim.
References
- Integration of multilayer regression analysis with structure-based pronunciation assessment
- Study of time and frequency variability in pathological speech and error reduction methods for automatic speech recognition
- On Using the UA-Speech and Torgo Databases to Validate Automatic Dysarthric Speech Classification Approaches
- This looks like that: deep learning for interpretable image recognition
- HyperPg - Prototypical Gaussians on the Hypersphere for Interpretable Deep Learning
- An Effective Local Prototypical Mapping Network for Speech Emotion Recognition
- This Looks Like That... Does it? Shortcomings of Latent Space Prototype Interpretability in Deep Networks
- Contrastive Prototypical Network with Wasserstein Confidence Penalty