Spending Listener Ratings Where the Quality Predictor Is Wrong

Judging synthetic speech still depends on human listeners. A panel rates each clip from 1 to 5, and the average is called the mean opinion score (MOS). Neural predictors that imitate these panels now agree closely with humans when scores are averaged over a whole system. On individual clips, and on systems unlike those in training, they agree much less. Practitioners are therefore left with two poor options: pay for a full listening test, or trust a predictor that never says when it is guessing. This proposal tests a middle path. We train a small model to forecast where the automatic predictor will be wrong on speech systems it has never seen. We then send only those clips to human listeners, and combine the two sources so the final system scores stay unbiased.

The claim we want to test is narrow. A predictor's confidence matters only if it ranks its own errors well under distribution shift, meaning when the speech comes from synthesis systems or languages the predictor was not trained on. We will build that error ranker as a light probe on the hidden layers of the speech model. We will train it on residuals from held-out systems, not in-distribution residuals. Under a fixed human-rating budget, it should beat both uniform spot-checking and standard variance estimates. If it does not, that is also a useful finding: uncertainty-guided triage would not be worth its complexity for MOS evaluation.

What exists and what is missing

Supervised MOS prediction is mature. MOSNet (Chen-Chou Lo et al.) framed it as regression from spectrograms. UTMOS (UTMOS) stacks self-supervised speech encoders (models pretrained on unlabeled audio, such as wav2vec 2.0) with listener embeddings and ranking losses, and won the VoiceMOS 2022 main track. The challenge report (The VoiceMOS Challenge 2022) and SOMOS (SOMOS) both document the same pattern: system-level agreement is high, while clip-level and cross-domain agreement is weaker. The cross-corpus numbers in Lifelong Learning MOS Prediction make the shift problem concrete. A model trained on Blizzard 2008–2013 reaches only 0.555 Spearman rank correlation (SRCC) on SOMOS. These papers motivate the work: the errors are real and concentrated off-distribution. None of them outputs a confidence.

Three papers bridge to uncertainty, and each leaves the delegation question open. FUSE-MOS predicts a Gaussian mean and standard deviation per clip, trained with negative log-likelihood. However, it never checks whether that standard deviation is calibrated. Its one use of the standard deviation, filtering uncertain training data, did not improve accuracy. We reuse its Gaussian head as a baseline, and we ask a different question: whether uncertainty can direct human effort at test time.

Uncertainty as a Predictor uses the output entropy of self-supervised models as a zero-shot quality signal, reaching about 0.70 SRCC. It finds that wav2vec 2.0 outputs are too invariant to degradation unless dropout is injected at inference. This result constrains our design: surface logits of strong encoders hide degradation. That is why we probe intermediate layers rather than outputs.

Calibration-Reasoning Framework for Descriptive Speech Quality Assessment uses "calibration" to mean supervised alignment of rating heads, not probabilistic confidence. It is relevant to us mainly through its finding that frozen encoders miss low-level degradation, gaining +0.12 Pearson correlation (PCC) when unfrozen. This is a warning that probing frozen layers may fail, and we test it directly.

The method draws on the calibration literature. Uncertainty Calibration for Tool-Using Language Agents shows that small probes on internal activations give far better calibration than surface probabilities: expected calibration error (ECE) of 0.035 versus 0.690 on TabMWP. Large Language Models Must Be Taught to Know What They Don't Know shows that asking a model to state its own confidence is brittle, with about 30% ECE. The delegation framing comes from When Do LLMs Need Retrieval Augmentation?, where reducing overconfidence lets a model call for retrieval on only about a third of queries without losing accuracy. That work routes to a retriever in text question answering. We route to human listeners in a regression task where the "correct answer" is itself a noisy panel average.

Core Claim

An error forecaster trained on leave-systems-out residuals of a strong MOS predictor will rank clip-level errors on unseen systems better than the predictor's own variance, ensemble disagreement, or Monte Carlo dropout. We expect the gap to be largest out of domain, where in-distribution uncertainty estimates are known to be overconfident (Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data). If that ranking is good, a fixed human budget allocated by the forecaster should reach the full-panel system ranking with substantially fewer ratings than uniform spot-checking.

Research Program

We first fix a base predictor: a UTMOS-style model fine-tuned on BVCC. We then build the error forecaster through cross-fitting. BVCC's 187 systems are split into folds. For each fold, a predictor is trained on the other folds and its absolute errors on the held-out systems are recorded. A small multilayer perceptron is then fit to predict those out-of-fold errors. Its inputs are pooled hidden states from several layers of the self-supervised encoder, plus the predictor's output, plus the Gaussian standard deviation where one is available. The forecaster therefore learns what "unfamiliar system" looks like in the encoder's internal representations, rather than learning noise patterns of the training set.

Because BVCC provides eight ratings per clip, we can separate two sources of uncertainty. One is listener disagreement, which more ratings average out. The other is predictor error, which they correct. We will report how much of the forecaster's signal tracks each.

The routing policy sends each clip to humans with a probability that increases with its forecast error, subject to a budget $B$ on the number of human ratings. The system score combines both sources with a weighted correction:

$$\hat{\mu}_s = \frac{1}{n_s}\sum_{i \in s}\Big(f(x_i) + \frac{h_i}{\pi_i}\,(y_i - f(x_i))\Big)$$

Here $f(x_i)$ is the machine score for clip $i$ of system $s$, $n_s$ is the number of clips in system $s$, $\pi_i$ is the probability that clip $i$ is sent to humans, $h_i$ is 1 if it was sent, and $y_i$ is the human mean for that clip. The estimate is unbiased for any choice of $\pi_i$. Its variance is lowest when $\pi_i$ grows with the true residual magnitude, so sampling well depends directly on how well errors can be forecast. Debiasing with a labeled subsample is a standard survey-statistics device. Our contribution is the speech-specific error forecaster and the empirical test of whether it beats uniform sampling.

All experiments simulate listening tests offline from existing ratings, so the human cost is counted in ratings consumed rather than collected anew.

Baselines

The do-nothing reference uses UTMOS predictions for every clip with zero human ratings. On BVCC it reports utterance-level MSE of 0.165 and SRCC of 0.897 (UTMOS). The challenge reports system-level SRCC around 0.939 for top systems (The VoiceMOS Challenge 2022). On the Chinese out-of-domain track, UTMOS reports utterance MSE of 0.162 and SRCC of 0.893. We retrain UTMOS under our system folds, so our absolute numbers on held-out systems will differ and will be reported side by side. The opposite reference, all clips rated by humans, defines the target ranking and the maximum cost.

The strongest real competitor is uniform random spot-checking combined with the same weighted correction. It is cheap, unbiased, and needs no uncertainty model, so any gain we claim must come from non-uniform allocation.

Three uncertainty-driven allocators share our routing and correction code and differ only in their score. The first is the FUSE-MOS Gaussian standard deviation (FUSE-MOS), retrained on our folds because the paper reports no numbers we can reuse. The second is the variance of a five-member deep ensemble. The third is Monte Carlo dropout variance, following the handicapping idea in Uncertainty as a Predictor. A heuristic that routes clips with predicted scores in the middle of the scale is also included.

Every method uses the same backbone, training data, folds, and tuning budget, with hyperparameters selected on a held-out validation fold of systems. The ensemble receives five times the training compute; we report this rather than hide it.

Two ablations isolate the key ingredient. The first trains the forecaster on in-distribution residuals instead of leave-systems-out residuals. The second feeds it only the final-layer output instead of intermediate hidden states. A third variant probes an unfrozen, fine-tuned encoder, which tests the frozen-representation warning from Calibration-Reasoning Framework. For probabilistic calibration of regression outputs on MOS benchmarks, we found no reported result on Lacuna; we will establish one.

Beating the baselines means two things. Our method must reach the full-panel system ranking at a given Kendall correlation using at least 30% fewer human ratings than uniform spot-checking, averaged over folds with confidence intervals. It must also show a lower area under the clip-level risk–coverage curve than the best variance baseline on held-out systems and on the Chinese out-of-domain track.

Why not just…?

Why not just use ensemble or dropout variance? This is the most likely way the proposal fails, and the experiment is designed to show it if so. The evidence against relying on variance is indirect. The forecaster's advantage should appear specifically on unseen systems, and variance methods are trained only on in-distribution data. In Mirror-Consistency, independent sampling left an ECE of 0.081 on GSM8K, versus 0.039 with arbitration. In Uncertainty as a Predictor, dropout helped but capped near 0.70 SRCC. These are signals from other settings, not proof for MOS. If ensemble variance matches the forecaster on held-out systems, we will report it as the recommended, simpler method.

Why not just apply conformal prediction to an existing predictor? Split conformal prediction gives guaranteed interval coverage only when test data are exchangeable with calibration data. Speech from new synthesis systems is exactly the case where that assumption fails. Conformal prediction also needs an uncertainty score to produce adaptive interval widths, and ours is a candidate for that score rather than a competitor. We will report the coverage of residual-normalized conformal intervals on held-out systems, using each uncertainty score in turn. This shows directly whether coverage survives the shift.

Why not just spot-check a random sample of clips? If predictor errors are essentially unpredictable, uniform sampling with the correction is optimal, and our forecaster is overhead. This is a real threat because listener noise dominates some clips. The budget curves against random sampling are the decisive comparison. The listener-disagreement decomposition will show how much residual error is forecastable at all.

Proposed Experiments

The primary setting is leave-systems-out cross-validation on BVCC. Transfer is tested by training on BVCC and evaluating on the VoiceMOS 2022 Chinese out-of-domain track and on SOMOS-clean. SOMOS-clean contains 200 neural TTS systems sharing one voice and vocoder, which stresses clip-level rather than system-level differences.

At the clip level, we measure four things. The first is how well each score ranks absolute error, via the area under the risk–coverage curve. The second is the continuous ranked probability score (CRPS), a proper score for predictive distributions, for methods that output distributions. The third is the empirical coverage of 80% and 90% intervals. The fourth is ECE computed on the binary event "error exceeds 0.5 MOS".

At the system level, we plot Kendall and Spearman correlation with the full-panel ranking against the number of human ratings spent. We also report the pairwise decision error rate: the fraction of system pairs ordered differently from the full panel at a fixed budget.

Three checks decide the outcome. The first is whether the leave-systems-out forecaster beats ensemble variance on held-out systems; if not, the probe is unnecessary. The second is whether non-uniform allocation beats uniform spot-checking at equal budget; if not, triage is unnecessary. The third is whether the advantage widens on the Chinese track and on SOMOS, as the shift hypothesis predicts, or shrinks there, which would mean the forecaster learned BVCC-specific patterns. A secondary analysis checks whether intermediate encoder layers predict error better than the final layer. This tests directly the observation in Uncertainty as a Predictor that the outputs of strong encoders are overconfident.