Borrowing a Protein Fitness Recipe to Score Speech Prosody

Automatic judges of synthetic speech usually score each acoustic channel on its own. They check whether the pitch is plausible, whether the words are intelligible, or how far the durations deviate. Phonetics says this is the wrong unit of analysis. In fast speech the vocal tract cannot reach its pitch targets, so a flattened tone can be correct for the timing it was produced with (Predicting taiwan Mandarin tone shapes from their duration). Emphasis likewise arises from pitch and articulation settling together into shared stable patterns, not from either channel alone (Dimensions of Prosodic Prominence in an Attractor Model). Protein fitness prediction faced a structurally similar problem and solved it cheaply. It had several partial views of each variant (a sequence model's opinion, a 3D-structure model's opinion), few lab measurements per protein, and a need for calibrated predictions. Kermut combines a zero-shot sequence score, used as the starting guess, with a sum of per-view similarity measures in a Gaussian process (a regression model that predicts with error bars from a similarity function between examples). It beats a large supervised transformer on ProteinGym in minutes rather than hundreds of hours (Kermut: Composite kernel regression for protein variant effects).

We propose to transfer that recipe to speech-quality prediction. The central change is where the interaction between channels lives. Pitch is scored conditionally on the realized phoneme durations. Pitch and duration then enter the regression through an explicit interaction term instead of an additive one. In two sentences: we first build a reference-free score that asks "is this pitch plausible given how fast these syllables were spoken?" We then use it as the prior mean of a composite Gaussian process that adapts to human ratings from a handful of labeled utterances and reports calibrated uncertainty.

Core Claim

Two things should hold together on the SOMOS listening-test benchmark. First, conditioning pitch plausibility on duration should be a better zero-shot predictor of human judgments than conditioning on phonemes alone. Second, a Kermut-style composite model built on that score should match or beat a fine-tuned neural opinion-score predictor when only a few dozen ratings per unseen synthesis system are available, and it should do so with calibrated intervals. We expect the gain to concentrate in utterances with unusual speaking rate. That is the regime where phonetics says channel-by-channel scoring misreads physical compression as an error.

What We Build On and Where We Depart

The zero-shot component extends TTScore (Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokens). TTScore trains sequence-to-sequence models on LibriSpeech-960 to predict discrete speech tokens from phonemes. One score uses HuBERT tokens for intelligibility. The other uses FACodec pitch tokens, pooled per phoneme, for prosody. It explicitly leaves duration, rhythm and energy unmodeled. We supply the missing piece, and we do so as a conditioning variable rather than as one more independent score.

The phonetic studies play a motivating role. The tone-shape and prominence studies show that interpreting pitch without timing or articulation is systematically misleading. The L2 work shows that accentedness and comprehensibility depend jointly on intonation, rhythm and rate (L1 Perceptions of L2 Prosody). None of these papers builds a predictor.

The protein side supplies the method. We reuse Kermut's structure: a zero-shot score serves as the prior mean, a learned weighted sum of kernels over modality-specific embeddings models what that score misses, and the regression is fit per target set from few labels. ProteinGym's per-assay supervised protocol gives us the evaluation template (ProteinGym). SSEmb adds a constraining lesson: an auxiliary channel helps most when the primary signal is thin. Structure lifted low-alignment-depth Spearman from 0.39 to 0.45 (SSEmb). We should therefore expect our interaction term to matter most on atypical-rate speech and little elsewhere, and we will report it that way.

Kermut's own limitations do not carry over. Its rigid-backbone and indel problems have no analogue here. Its cubic scaling does, and we address it below.

Research Program

The zero-shot scorer has two factors.

The first is a duration model. It gives $p(d \mid x)$, the probability of the per-phoneme duration sequence $d$ (obtained by forced alignment) given the phoneme sequence $x$.

The second is a pitch model. It gives $p(z \mid x, d)$, the probability of the per-phoneme FACodec pitch tokens $z$ given both phonemes and realized durations. We train it alongside a TTScore-style model $p(z \mid x)$ that does not see durations. All three are trained on LibriSpeech-960. Energy enters as a third token stream conditioned the same way.

We keep the factors as separate features rather than a single joint likelihood. This lets the downstream model tell "the timing is odd" apart from "the pitch is odd for this timing." The gap $\log p(z \mid x,d) - \log p(z \mid x)$ is directly interpretable: it is how much of the apparent pitch error the timing explains.

The second stage is the composite regression. For utterance $u$, the predicted listener score is a Gaussian process with mean $m(u) = a + b\cdot s(u)$, where $s(u)$ is the zero-shot score from stage one. Its covariance is

$$k(u,u') = \pi_1 k_z(u,u') + \pi_2 k_d(u,u') + \pi_3 k_z(u,u')\,k_d(u,u') + \pi_4 k_h(u,u'),$$

where $k_z$, $k_d$ and $k_h$ are similarity kernels on phoneme-pooled pitch, duration and HuBERT content embeddings, and the weights $\pi_i \ge 0$ are learned. The product term $k_z k_d$ says two utterances are similar only if they share both pitch pattern and timing. It is the regression-side expression of prosodic interaction.

Fitting will be cheap because each few-shot problem has at most a few hundred labeled points. For pooled experiments over the 20,000 SOMOS samples, we use inducing-point approximations to avoid the cubic cost. Embeddings come from existing speech encoders, for example LightHuBERT.

Baselines

The naive reference is the predictor that outputs each system's mean rating from its labeled shots, together with word error rate as a single-channel metric. TTScore reports WER at an utterance-level Pearson correlation of 0.32 on SOMOS.

The strongest reference-free competitor is TTScore itself, run zero-shot. Its intelligibility score reaches utterance-level Pearson 0.53 on SOMOS (Objective Evaluation of Prosody and Intelligibility…). We found no reported SOMOS number on Lacuna for TTScore-pro alone. We will retrain both TTScore components on the same LibriSpeech-960 data and token extractors so that the only difference from our scorer is the duration conditioning. Beating it zero-shot means a higher utterance-level Pearson and Spearman correlation than the best TTScore combination.

The strongest supervised competitor is a fine-tuned self-supervised opinion-score predictor: a HuBERT-family encoder with a regression head, trained on SOMOS training systems and then fine-tuned on the same $k$ labeled shots per held-out system. We found no reported result on Lacuna for such a predictor on SOMOS, so we will tune it ourselves. It gets the same hyperparameter search budget as our model, and we report its compute. Beating it means equal or better correlation at $k \le 50$ plus better negative log-likelihood and interval coverage. Equal correlation with better calibration would still be a meaningful result.

Three ablations isolate the key ingredients. The first replaces the conditional pitch score with the phoneme-only score, which isolates duration conditioning. The second drops the product kernel term, giving an additive model that isolates the interaction. The third gives a multilayer perceptron the identical features and prior, which isolates the Gaussian process machinery from the features.

Why not just…?

Why not just fine-tune a neural opinion-score predictor on more ratings? This is the most serious threat. With thousands of ratings from the target system, a fine-tuned encoder may well win on correlation. Our claim is confined to the few-label, new-system regime, which Kermut-versus-ProteinNPT suggests favors kernel methods (Kermut). We will plot both methods against $k$ from 0 to several hundred and report where the curves cross. If they never cross in our favor, the composite-model half of the claim fails.

Why not just concatenate pitch, duration and energy features into a flexible regressor, which can learn interactions on its own? It can, given enough data. The multilayer-perceptron ablation receives exactly our features and prior. If it matches the Gaussian process at small $k$ and is calibrated after post-hoc temperature scaling, the explicit kernel product adds nothing, and we will say so.

Why not just add duration as another token stream in TTScore and score the joint likelihood? A joint likelihood is mathematically the product of our two factors. Summing them into one number throws away which channel is responsible. The first ablation, plus a variant that feeds only the summed joint score, tests whether keeping the factors separate actually helps prediction. If it does not, the decomposition remains useful only as a diagnostic.

Proposed Experiments

The primary experiment holds out SOMOS synthesis systems. Models are trained or fit on the remaining systems, and each held-out system contributes $k \in \{0, 10, 25, 50, 200\}$ labeled utterances. We report utterance- and system-level Pearson and Spearman correlation, negative log-likelihood, and 90% interval coverage, averaged over repeated draws of the shots.

The decisive check is stratified. We bin test utterances by speaking rate relative to the phoneme-conditioned duration model. The prediction is that the conditional-pitch gain and the interaction-kernel weight $\pi_3$ are largest in the fast and slow tails and near zero in the middle. A uniform gain, or none, would contradict the phonetic motivation even if the overall numbers improve.

A secondary check uses TTSArena pairwise preferences. We test whether our scores predict which of two systems listeners prefer. We found no reported Lacuna number to compare against, so this is reported as exploratory.

Finally, as a sanity check on the transfer itself, we inspect the learned $\pi_i$. If the content kernel absorbs nearly all the weight, prosody is not what the ratings track, and the proposal reduces to a cheaper intelligibility predictor.