Automatic voice-quality grading today learns from expert perceptual ratings. Clinicians listen to each recording and score it on a standard scale such as the Consensus Auditory-Perceptual Evaluation of Voice (CAPE-V). These ratings are slow to collect and vary between raters, and they give one number per recording, which says nothing about where in a sentence the voice breaks, strains, or turns breathy. Far cheaper labels already exist in clinical databases: a recording-level diagnosis of "healthy" or "pathological." We propose to train a continuous voice-severity scorer from those diagnosis-only labels. The model scores every short frame of connected speech and then combines the frame scores with attention, a learned weighting that decides how much each frame counts toward the recording-level score. A handful of expert ratings, about twenty, are used only to fix the scale of the output and never to set its ordering.
The core claim is specific. If weakly supervised attention pooling is given two constraints, it can rank connected-speech recordings by severity well enough to approach a model trained on full CAPE-V ratings, and its frame scores will mark the intermittent events that drive perceived severity. The first constraint ties frame scores to the diagnosis: no frame of a healthy recording may look severe, and some frames of a pathological one must. The second, "detachable" pooling, prevents the attention module from hiding corrupted frame features. The frame-level map is something current rating-supervised systems do not produce, and the label cost drops from consensus perceptual ratings to diagnoses that are already in the chart.
Why this gap, and what constrains the design
The motivating system is VOQANet+. It learns a weighted mix of all transformer layers of speech foundation models (WavLM, HuBERT, Whisper), adds jitter, shimmer and harmonics-to-noise ratio, and pools over time with attention trained on averaged CAPE-V ratings. It reaches a Pearson correlation of 0.908 on the sentence subset of the Perceptual Voice Quality Database (PVQD-S). It needs continuous expert ratings, collapses rater disagreement into a mean, and offers no supported setting where only diagnoses are known. We reuse its feature stack unchanged and replace its supervision.
The weak-label recipe borrows from imaging. Weakly supervised ulcerative colitis severity estimation trains frame-level endoscopy graders from a single patient-level "maximum severity" score. Its triplet loss enforces that no frame exceeds the patient maximum, and it reaches 71.4% accuracy against 52.1% for unsupervised adaptation. The authors note that this loss depends on a strictly ordinal four-level score. We adapt the idea to a binary diagnosis paired with a continuous frame score, which their formulation does not cover. Whole-slide pathology work frames the same structure as multiple instance learning, meaning learning from labels on bags of unlabeled pieces. TransMIL models correlations between patches. ADR/AEM shows that slide-label training makes attention collapse onto a few patches, and fixes this by maximizing attention entropy early in training. AEM is formulated only for classification and spreads attention blindly. Both serve as competitors here, not as the method.
Two results from speech constrain the design. Wang et al. found that in attentive speaker embeddings the attention tracks more than voice activity: its correlation with a separate voice-activity detector was only r = 0.37, and the weights transferred to an unrelated i-vector system, cutting its error by 9.0%. That is encouraging, because attention carries real frame-importance information. The same paper also found that features trained under attention degrade badly when pooled uniformly, with EER above 18%. The encoder had learned to rely on attention to discard frames it left corrupted. For our purpose this is the central danger. If low-attention frames carry poisoned features, their frame scores are meaningless, and any "localization" is a mirage. Separately, Huang and Narayanan found that deeper attention scorers overfit on small emotion corpora. Their attention-based model gave only a modest gain over uniform pooling on IEMOCAP, 59.33% against 57.87% weighted accuracy. We therefore keep the scorer shallow.
Core mechanism
Let $h_t$ be the layer-weighted foundation-model feature of frame $t$ in a recording of $T$ frames. A linear frame head gives a severity $s_t = \sigma(w^\top h_t) \in [0,1]$. A one-layer attention scorer gives weights $\alpha_t$ that sum to one. The recording score is $\hat{y} = \sum_t \alpha_t s_t$. With diagnosis $d \in \{0,1\}$ and a top-$k$ mean $\mathrm{top}_k(s)$ over the $k$ highest frame scores, the training loss is
$$ \mathcal{L} = \mathrm{BCE}(\hat{y}, d) + \lambda \Big[ (1-d)\,\max(0,\ \max_t s_t - \tau) + d\,\max(0,\ m - \mathrm{top}_k(s)) \Big], $$
where $\tau$ is a ceiling that no healthy frame may exceed, $m$ is a floor that the most severe pathological frames must reach, and $\lambda$ weights the constraint. The healthy-side term is the continuous analog of the maximum-severity rule. The pathological-side term stops the model from explaining a diagnosis with diffuse, low-level scores.
Detachable pooling handles the poisoned-feature problem. On a random fraction $p$ of training steps, $\alpha$ is replaced by uniform weights. This forces every frame's $s_t$ to be individually meaningful, because the head cannot rely on attention to hide frames. At test time the recording score uses learned attention.
Finally, about twenty CAPE-V-rated recordings fit an isotonic map, a monotone curve, from $\hat{y}$ to the CAPE-V scale. The map cannot change rank order, so any rank correlation with CAPE-V comes entirely from diagnosis labels.
Research Program
The main setting uses genuine diagnosis labels. We train on the Saarbrücken Voice Database (SVD), which has over 2,000 German speakers labeled healthy or pathological, using its connected-speech recordings where available and sustained vowels otherwise. We then evaluate severity ranking against CAPE-V on English PVQD and PVQD-S. This reverses VOQANet+'s cross-database direction: it trained on PVQD ratings and tested on SVD diagnoses. The second setting is a controlled simulation. On PVQD-S itself, we withhold the continuous ratings and keep only a thresholded diagnosis-like label, with speaker-disjoint folds. This isolates the supervision change from the language and domain change. It mildly favors the weak model, because the threshold derives from the ratings, and we report it as such.
Localization is the part rating-supervised systems cannot be graded on, so we build a small evaluation set. A speech-language pathologist marks voice breaks, strained spans and breathy spans on about 50 PVQD-S sentences, roughly an hour of expert time. This is far less than producing consensus ratings for a training corpus. We also report deletion curves: remove the top-scored frames and measure how much the recording score falls, compared with removing random frames or frames of equal energy. The energy control checks that the model is not just a loudness detector. We further check how frame scores correlate with locally computed jitter, shimmer and harmonics-to-noise ratio, as a weak physical sanity check.
Baselines
The naive reference is the simplest weak-label model: a mean-pooled VOQANet-style layer-weighted embedding, a logistic classifier trained on the same diagnoses, and the classifier's logit used as the severity score after the same isotonic calibration. It uses the same features, data, calibration set and tuning budget, so the only difference from our method is the pooling and the constraints. Beating it means a higher Spearman and Pearson correlation with CAPE-V on PVQD-S across five seeds with non-overlapping bootstrap intervals, and frame maps that beat it on clinician-span overlap.
The strongest existing method is VOQANet+ trained on full PVQD-S CAPE-V ratings, which reports PCC 0.908. It sees far more expensive labels, so it is an upper reference, not a competitor we expect to beat. Success means closing most of the gap between the naive baseline and 0.908 while using only diagnoses plus twenty calibration ratings. We will retrain it under our folds rather than rely on the reported figure alone.
The label-matched competitor is VOQANet+ trained by regression on the same twenty rated recordings we use for calibration. It answers whether a few direct ratings beat many diagnoses.
The weak-supervision competitor is attention-based multiple-instance pooling with AEM entropy annealing. It is trained on the same diagnoses, with its entropy weight tuned on the same budget as our $\lambda$, $\tau$, $m$ and $p$.
Two ablations isolate our ingredients. The first removes the frame constraints. The second removes detachable pooling. In the second, we expect recording-level correlation to hold up while localization and uniform-pooling accuracy collapse, reproducing the poisoned-feature effect.
Why not just…?
Why not just use the mean-pooled diagnosis classifier's logit as the severity score? This is the real threat. On sustained vowels, where dysphonia is fairly uniform across the recording, it may match us, and we expect it to. Our bet is limited to connected speech, where severity often comes from brief breaks and strain episodes that averaging dilutes. That is the same dilution argument that motivated attention in emotion and speaker work. If the logit equals our model on PVQD-S ranking and on clinician-span overlap, the method has no reason to exist, and we will report that.
Why not just collect a few ratings and fine-tune VOQANet+ directly? Twenty recordings is a tiny regression set for a model mixing three foundation encoders. The emotion-corpus overfitting result from Huang and Narayanan suggests it will be unstable. We have no reported number for this few-shot setting on Lacuna. The label-matched baseline measures it directly, and we will also sweep the rating budget from 20 to 200 to find the point where direct ratings overtake diagnoses.
Why not just add AEM entropy regularization? AEM addresses collapse of attention onto a few patches, but nothing in it ties frame scores to the diagnosis, and nothing stops low-attention frames from carrying poisoned features. It could match us on recording-level ranking. Our expectation is that it falls behind on localization and on the uniform-pooling test, and the head-to-head comparison settles this.
Decisive checks
The proposal succeeds if three things hold together. On PVQD-S, the SVD-trained model's rank correlation with CAPE-V clearly exceeds the mean-pooled logit and AEM. Its frame scores overlap clinician-marked spans better than attention from those baselines, and better than energy-matched random frames on deletion tests. Removing detachable pooling degrades localization far more than recording-level correlation. It fails cleanly if the mean-pooled logit ties it on connected speech. That outcome would itself be useful, because it would mean diagnosis labels already suffice and attention adds nothing for severity ranking.
References
- Towards Robust Assessment of Pathological Voices via Combined Low-Level Descriptors and Foundation Model Representations
- Domain Adaptation for Ulcerative Colitis Severity Estimation Using Patient-Level Diagnoses
- TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification
- ADR: Attention Diversification Regularization for Mitigating Overfitting in Multiple Instance Learning based Whole Slide Image Classification
- Attention Mechanism in Speaker Recognition: What Does it Learn in Deep Speaker Embedding?
- Attention Assisted Discovery of Sub-Utterance Structure in Speech Emotion Recognition