Are Zero-Shot Voice Cloners Cloning, or Remembering? A Speaker-by-Text Contamination Audit

Zero-shot voice cloning systems take a few seconds of audio from a speaker and read new text aloud in that voice, without retraining. The best results in the field come from systems trained on tens of thousands of hours of LibriVox audiobooks and then tested on LibriSpeech, which is a benchmark cut from the same LibriVox audiobooks. Two things can therefore leak from training into the test. The first is the narrators, since the same volunteers record many books. The second is the sentences, since roughly two-thirds of LibriSpeech's test text already appears in common text pretraining corpora. Nobody has measured how much of the reported "zero-shot" quality comes from this overlap. Some of it may be memory of familiar voices and familiar sentences rather than generalization to new ones.

We propose a small, cheap audit that separates the two kinds of leakage. The design crosses seen and unseen speakers with seen and unseen text. Unseen text is generated procedurally, meaning sentences are produced by a controllable generator so they cannot exist anywhere online. Unseen speakers come from LibriVox narrators whose first recordings were released after a model's training corpus was frozen. These speakers are in the same audiobook domain but provably absent from training. The audit runs as a modular harness on an academic budget, and every system gets the same decoding-tuning budget so that tuning luck is not mistaken for generalization. The result is a measured "contamination gap" for each system, plus a controlled training experiment that tests whether such a gap is causal.

Core Claim

The core claim has three parts. First, standard LibriSpeech test-clean numbers for zero-shot cloning mix three things: generalization to new voices, familiarity with narrators heard in training, and familiarity with sentences seen in training. Second, a two-by-two design (speaker seen or unseen, text seen or unseen) holds the recording domain fixed, so it can pull these three apart. Third, there is a measurement problem the field has not noticed. Intelligibility in this literature is scored by running an automatic speech recognizer over the synthesized audio. That recognizer can itself be biased toward leaked sentences, so word error rate on LibriSpeech text may be optimistic for every system at once, ground truth included.

Each part motivates a different piece of the design. The first rests on Evaluation of LLMs in Speech is Often Flawed. That paper matched LibriSpeech and Common Voice transcripts against The Pile, found about two-thirds and one-third leaked respectively, and showed that language models trained with leakage give leaked sentences lower perplexity, which biases recognition toward them. It studied only recognition, never synthesis or speaker identity. We carry its controlled-contamination method over to voice cloning, and its finding is also the basis for our worry about the recognizer used as a metric.

The flagship systems show where the overlap sits. VALL-E recasts speech synthesis as next-token prediction over discrete audio codes and trains on LibriLight (60,000 hours from over 7,000 speakers). VALL-E 2 trains on Libriheavy (50,000 hours, about 7,000 speakers) and reports human parity on LibriSpeech. VALL-E R adds forced monotonic alignment and reports near-ground-truth word error rate on LibriSpeech. All three draw training and test data from the same parent collection, and none reports a speaker-overlap check based on the audio itself.

Research Program

The work has four components.

A modular harness. We separate data iteration, synthesis, post-processing and metric computation, following the four-stage decomposition of OmniEvaluator. That system cuts integration cost by routing every model and benchmark through one intermediate format, but it supports only comprehension tasks, and its authors state that their judge cannot assess speaker identity or speech naturalness. We add the missing generative-audio metrics as plug-in scorers:

  • speaker similarity, computed as the cosine between speaker-verification embeddings of the reference and the synthesized audio;
  • intelligibility, measured as word error rate from more than one recognizer;
  • a small human similarity-rating study.

The point of the modularity is narrow and practical. The same test cell must reach different synthesizers and different scorers without per-model glue code, which DeepDIVA identifies as a common source of irreproducible results. The decomposition is borrowed; the audio scorers and the factorial data module are new.

An acoustic overlap audit. For each LibriSpeech dev and test speaker, we search LibriLight and Libriheavy for near-duplicate voices. The search uses nearest neighbours in speaker-embedding space, with a match threshold calibrated on known same-reader and different-reader LibriVox pairs. This catches overlap that metadata checks miss, for example a narrator who appears under a different reader identifier or whose recordings were re-segmented. We report it next to a plain reader-ID metadata match, so the added value of the acoustic check is itself measured.

The two-by-two test set. The speaker factor uses two groups:

  • seen speakers: LibriSpeech test speakers flagged by the audit;
  • unseen speakers: narrators whose first LibriVox release postdates the corpus snapshot, verified against the same audit. This borrows the post-cutoff logic that Unveiling Causal Reasoning in Large Language Models used for events after a training cutoff.

The text factor also uses two groups:

  • seen text: LibriSpeech transcripts found in The Pile;
  • unseen text: sentences produced by a phonetically controlled grammar, in the spirit of DyVal. DyVal generates reasoning problems from graphs with adjustable difficulty. We generate sentences with adjustable length, density of rare phoneme pairs, and syntactic depth, and we match them to the seen text on length and on phoneme distribution.

The reference prompt is always real audio from the target speaker, and the target is a sentence that speaker never recorded. Defining $m_{st}$ as a metric's mean value in the cell with speaker status $s$ and text status $t$, each taking the value seen or unseen, the speaker leakage effect is

$$\Delta_{\text{spk}} = m_{\text{seen},t} - m_{\text{unseen},t},$$

averaged over both text conditions. The text effect $\Delta_{\text{txt}}$ is defined the same way across text status. The interaction term tells us whether the two kinds of leakage compound.

A controlled causal test. The audit on public checkpoints is correlational. To test cause, we train small codec language models on LibriTTS, which has 585 hours from 2,456 speakers (LibriTTS). We train two variants: one with a chosen set of evaluation speakers injected at controlled amounts of audio, and one without. This mirrors the from-scratch contamination design of the speech recognition paper and fits a single-node budget.

An equal tuning budget. Every system receives the same small random search over its inference settings, using a dev split drawn only from the unseen cells:

  • sampling temperature and top-p for the autoregressive models;
  • the repetition-window and threshold settings of VALL-E 2's repetition-aware sampling, where a reimplementation exposes them;
  • OpenVoice's style and flow-temperature knobs.

We report the gap both at default settings and at the tuned setting. This follows the warning in Hyperparameter Optimization Under Strict Computational Budgets and Jargon: when tuning is asymmetric, comparisons are confounded. The equal-budget rule exists to prevent that, not to win benchmarks.

Baselines

The naive reference. This is what the field reports today: one score on LibriSpeech test-clean at default decoding settings. For intelligibility, the strongest reported figure is from VALL-E R: word error rate of 1.58% on LibriSpeech continuation, against 2.37% for VALL-E and 1.41% for ground truth. For similarity, VALL-E 2 reports a similarity rating of 4.61 against 4.13 for ground truth. No reported result was found on Lacuna for embedding-cosine speaker similarity on LibriSpeech, so our harness will establish that reference itself. We do not need to reproduce these exact numbers. The point is that our test-clean cell recreates the naive protocol inside the harness, under the same scorers as every other cell.

The strongest existing evaluation practice. This is static out-of-domain testing on VCTK, as used by VALL-E and VALL-E 2, combined with reader-ID metadata deduplication. We run it on identical systems, scorers and tuning budget. Our protocol beats it if it detects a significant speaker or text gap that VCTK-plus-metadata either misses or misattributes to domain shift. A second way to beat it is if our unseen-in-domain cell reorders systems relative to the VCTK ranking.

The systems under audit. These are OpenVoice and publicly available codec language models trained on LibriVox-derived corpora. OpenVoice splits cloning into a base speaker model, which controls rhythm and style, and a tone-colour converter that swaps in the target timbre; its training uses a 20,000-speaker multilingual set. It is the natural foil because its timbre path is a narrow speaker embedding and should be less able to memorize individual narrators.

Ablations. Two ablations isolate the key ingredients. The text-only arm (seen speakers, procedural text) and the speaker-only arm (post-cutoff speakers, leaked text) each remove one leakage channel. We also re-score every cell with a recognizer whose language model is swapped for an acoustic-only decoder, which isolates metric contamination from synthesis contamination.

Why not just…?

Why not just evaluate on VCTK, which is already out of domain? VCTK changes speaker status and recording domain at once. It uses studio recordings of accented read speech, while LibriVox is home-recorded audiobook narration. A drop on VCTK therefore cannot distinguish "the model memorized LibriVox narrators" from "the model struggles with a new channel and new accents." The post-cutoff LibriVox arm holds the domain fixed, which is exactly what VCTK cannot do. We run VCTK as a baseline, so the comparison is shown directly rather than assumed.

Why not just deduplicate on LibriVox reader IDs? This is a real threat, because it is cheap and may catch most of the overlap. The audit reports both the metadata match and the acoustic match. If the acoustic search finds essentially nothing beyond the metadata, we will say so and recommend the cheap check. Even then, the two-by-two design is still needed, because metadata cleaning says nothing about leaked text or about the recognizer bias.

Why not just record a fresh set of new speakers reading new text? That is the cleanest possible unseen cell, and we include a small one as an anchor. It cannot by itself separate speaker leakage from text leakage, and it is expensive to scale. If the fresh anchor and the post-cutoff LibriVox cell agree, the scalable cell is validated. If they disagree, post-cutoff narrators are not as clean as assumed. Either result is informative.

Why not just use default decoding settings? VALL-E 2 documents that plain sampling falls into repetition loops. That means default settings could create an apparent gap on the harder unseen cells that disappears once decoding is tuned. Reporting the gap at both default and equal-budget tuned settings shows whether this happens.

Proposed Experiments

First decisive check: does contamination cause a gap? In the controlled LibriTTS experiment, injected speakers should show higher speaker similarity than matched held-out speakers. The gap should be larger than the spread across random seeds, and it should grow with the amount of injected audio.

  • If it does not grow, the memorization hypothesis is unsupported at this scale, and we report the negative result.

Second decisive check: do the public checkpoints show the gap? We estimate $\Delta_{\text{spk}}$ and $\Delta_{\text{txt}}$ on public checkpoints with bootstrap confidence intervals.

  • We predict a positive speaker gap for the codec language models trained on LibriVox, and a smaller one for OpenVoice's narrow timbre path.
  • If both systems show equal gaps, that prediction is falsified.

Third check: is the metric itself contaminated? We look for word error rate on leaked text being lower for ground-truth audio as well as for synthesized audio, compared with matched procedural text. The difference should shrink under the acoustic-only recognizer. That pattern would show that part of the reported intelligibility advantage belongs to the scorer, not the synthesizer.

Fourth check: does the ranking change? We compare the ordering of systems under three protocols: naive test-clean, VCTK, and the unseen-in-domain cell, all at equal tuning budget. A change in ordering is the result that would matter most to practitioners.

Budget. The whole study stays within a fixed budget: a single-node budget for training, a fixed random-search size per system, and a human rating study limited to a few hundred items concentrated where the automatic scores disagree.