Is a TTS Win on One Voice a Win at All? Speaker as a Random Effect in Listening Tests

Most claims that one text-to-speech system sounds better than another rest on a listening test run on a single voice, often the female reader in LJSpeech. The p-value in such a paper answers a narrow question: would these listeners, hearing these sentences in this one voice, prefer system A again? Readers take it as a broader claim, that A beats B for voices in general. These are different statistical questions. The gap between them is the gap a physicist sees between a local fluctuation and a result that survives independent replication. We propose to measure that gap directly. We will run crossed listening tests in which speakers, listeners and sentences are all treated as random samples. We then estimate how much a system's advantage changes from one voice to another, and from that derive how a fixed listening budget should be divided among speakers, listeners and sentences.

The core claim is simple, and as far as the material we found shows, it has not been tested. The speaker-by-system interaction, meaning the degree to which A's advantage over B depends on which voice is synthesized, is a variance component large enough that single-speaker significance often fails to replicate on new voices. The second part of the claim is that a model fit to individual ratings, which respects their ordinal scale, can report "generalizes across speakers" separately from "significant on this voice."

What prior work established and left open

The direct precedent is Bias and Statistical Significance in Evaluating Speech Synthesis with Mean Opinion Scores. It showed that listener offsets (an average 2.31-point spread in mean listener scores) and sentence difficulty (a 1.47-point spread) blur system comparisons across 46 crowdsourced tests. It also showed that rank-normalizing scores per listener and per sentence before a Mann-Whitney test raised the test statistic in 41 of 46 tests. That paper motivates this proposal and supplies part of the method. Our proposal differs in two ways.

First, it held the target voice fixed, so it could not ask whether results carry over to other voices. Second, normalization removes each listener's or sentence's average offset. It cannot represent an interaction, where the system difference itself varies by speaker. Our approach is built around that interaction.

The case that speaker matters comes from several directions. Prosodic Representation Learning and Contextual Sampling for Neural Text-to-Speech reports that prosody learned from one narrator does not transfer across speaking profiles. Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention finds that single-speaker synthetic augmentation lacks the acoustic variety needed for real deployment. Effects of Speaker Count, Duration, and Accent Diversity on Zero-Shot Accent Robustness in Low-Resource ASR shows that for training speech recognizers, spreading a fixed budget over more speakers beats more hours per speaker. We ask the evaluation-side version of that question: should a fixed number of listener judgments be spread over more voices or more sentences?

Intra-speaker phonetic variation in read speech adds a complication. The acoustic features that best distinguish one speaker from another, such as mean pitch and low-frequency energy, also drift within a single speaker across recording sessions. So "speaker" may itself need to be split into a speaker component and a session component.

From the physics literature we borrow two lessons as a methodological lens rather than as technique. Results reported at intermediate significance are labeled "evidence" pending independent replication, as with the 3.4σ vector state in Evidence for a vector charmoniumlike state…. Re-examination of the test statistic can also dissolve an apparent effect, as Neutrino-4 anomaly: Oscillations or fluctuations? shows. The unbinned-likelihood tradition supplies a second lesson. Averaging ratings into a mean opinion score (MOS) before testing is a form of binning that discards information. We instead fit a likelihood to every individual rating.

Core Claim and Model

We model each 1–5 rating with a cumulative-link mixed model. This is an ordinal regression in which a latent score, crossed with fixed cut-points, produces the observed category, so the scale is never treated as interval data. The latent score includes a fixed system effect plus random intercepts and random system slopes for speaker, session-within-speaker, listener and sentence. Under this model the variance of the estimated system difference is approximately

$$ \mathrm{Var}(\hat\Delta) \approx \frac{\sigma^2_{S\times\text{sys}}}{S} + \frac{\sigma^2_{L\times\text{sys}}}{L} + \frac{\sigma^2_{U\times\text{sys}}}{U} + \frac{\sigma^2_{\varepsilon}}{n}. $$

Here $S$, $L$ and $U$ are the numbers of speakers, listeners and sentences, $n$ is the total number of ratings, each $\sigma^2_{\cdot\times\text{sys}}$ is the variance of the system difference across that factor, and $\sigma^2_\varepsilon$ is the residual variance. The form follows from averaging independent crossed random effects.

The key point is the case $S=1$. A single-voice test cannot estimate $\sigma^2_{S\times\text{sys}}$ at all, and no number of listeners can shrink the first term. Each test therefore gets two separate outputs: significance conditional on the tested voices, and significance for the population of voices.

Research Program

We will build a crossed rating corpus. Ten system pairs are drawn from multi-speaker adaptation methods: FastPitch with bottleneck adapters as in Adapter-Based Extension of Multi-Speaker Text-To-Speech Model for New Speakers, the meta-learned FastSpeech 2 adaptation of Meta-TTS, and variants differing in a single design choice, such as decoder size or amount of adaptation data. Each pair is adapted to 40 LibriTTS speakers plus the single-speaker corpora LJSpeech, RyanSpeech and, for French, SIWIS.

To separate session drift from speaker identity, we will also adapt to the nine PATAFreq speakers using recordings from different sessions. Listeners rate in a balanced incomplete design so that every factor is crossed. From this corpus we estimate the variance components, then simulate from the fitted model to produce power curves over $(S, L, U)$ at a fixed total number of ratings.

Baselines

The naive reference point is what most papers report: a Student's t-test on MOS for one voice. This is the practice the MOS-bias paper criticizes.

The strongest existing method is that paper's procedure: rank-normalize scores per listener and per sentence, then apply a Mann-Whitney test. It found 24 of 46 tests significant at p<0.01 with the plain Mann-Whitney test, and 26 after joint normalization. We run it on exactly the same ratings. On the multi-speaker data we pool all voices, which is the natural extension.

We add a third baseline that adds per-speaker rank normalization. This tests whether removing speaker offsets is enough.

The key ablation is our own mixed model with the speaker-by-system slope deleted. It isolates whether the interaction term, rather than the ordinal likelihood or the per-rating fit, accounts for any difference.

For mixed models of this kind on TTS ratings, no reported result was found on Lacuna, so we have no prior number to beat. Every method gets the same ratings and no tuning beyond fixed significance levels.

Our model counts as beating the baselines on two conditions. It must hold nominal false-positive rates in A/A tests where the baselines exceed them. It must also predict the sign of a system difference on held-out speakers more accurately, at equal or better power.

Why not just…?

Why not just evaluate on several speakers and pool the ratings? Pooling gives every rating equal weight and treats voices as fixed. The resulting p-value still ignores how much the system difference varies across voices, so it overstates confidence whenever $\sigma^2_{S\times\text{sys}}$ is nonzero. The pooled Mann-Whitney baseline is exactly this alternative. The A/A false-positive comparison and the held-out-speaker replication check will show whether its overconfidence matters in practice. If the estimated interaction turns out to be negligible, pooling is adequate, and we would report that as the finding.

Why not just extend rank normalization to speakers? Normalizing per speaker removes each voice's average quality but leaves untouched the part where A wins on one voice and loses on another. That makes tests look more significant while answering the narrower, fixed-voice question. Our third baseline implements it, and we expect it to have the highest power and the worst false-positive rate on held-out speakers.

Why not just use objective metrics such as speaker-embedding similarity or recognizer word error rate? These avoid listener noise but measure intelligibility and timbre match, not naturalness. They also vary by speaker and need the same random-effects treatment. We compute them on all samples and report whether they track the human interaction structure. They are a complement, not a substitute.

Proposed Experiments and Decisive Checks

The first decisive check is calibration. We build A/A pairs, meaning the same system trained with two seeds, and require each method's false-positive rate at α=0.01 to stay near 1% when the conclusion is applied to new speakers.

The second is replication. For each system pair we fit on LJSpeech alone, then on random single LibriTTS voices, and score how often a significant single-voice result has the same sign and remains significant on the other 39 voices. This replication rate is the headline number.

The third uses the PATAFreq sessions to test whether session-within-speaker variance is comparable to speaker variance. If it is, a "speaker" in an evaluation is not one fixed unit, and adapting a voice from one recording session understates the variability.

Finally, from the fitted variance components we publish a budget rule: the number of voices, listeners and sentences that minimizes $\mathrm{Var}(\hat\Delta)$ for a given number of ratings. We validate it by subsampling the real corpus. The claim fails if the speaker-by-system variance is small relative to the listener and sentence terms, in which case single-voice tests are vindicated.

References

  1. Bias and Statistical Significance in Evaluating Speech Synthesis with Mean Opinion Scores
  2. Prosodic Representation Learning and Contextual Sampling for Neural Text-to-Speech
  3. Detection of Lexical Stress Errors in Non-Native (L2) English with Data Augmentation and Attention
  4. Effects of Speaker Count, Duration, and Accent Diversity on Zero-Shot Accent Robustness in Low-Resource ASR
  5. Intra-speaker phonetic variation in read speech: comparison with inter-speaker variability in a controlled population
  6. Evidence for a vector charmoniumlike state in <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msup><mml:mi>e</mml:mi><mml:mo>+</mml:mo></mml:msup><mml:msup><mml:mi>e</mml:mi><mml:mo>−</mml:mo></mml:msup><mml:mo stretchy="false">→</mml:mo><mml:msubsup><mml:mi>D</mml:mi><mml:mi>s</mml:mi><mml:mo>+</mml:mo></mml:msubsup><mml:msubsup><mml:mi>D</mml:mi><mml:mrow><mml:mi>s</mml:mi><mml:mn>2</mml:mn></mml:mrow><mml:mo>*</mml:mo></mml:msubsup><mml:mo stretchy="false">(</mml:mo><mml:mn>2573</mml:mn><mml:msup><mml:mo stretchy="false">)</mml:mo><mml:mo>−</mml:mo></mml:msup><mml:mo>+</mml:mo><mml:mi mathvariant="normal">c</mml:mi><mml:mo>.</mml:mo><mml:mi mathvariant="normal">c</mml:mi><mml:mo>.</mml:mo></mml:math>
  7. Neutrino-4 anomaly: Oscillations or fluctuations?
  8. Adapter-Based Extension of Multi-Speaker Text-To-Speech Model for New Speakers
  9. Meta-TTS: Meta-Learning for Few-Shot Speaker Adaptive Text-to-Speech
  10. RyanSpeech: A Corpus for Conversational Text-to-Speech Synthesis
  11. The SIWIS French Speech Synthesis Database ? Design and recording of a high quality French database for speech synthesis