Zero-shot voice cloning asks a text-to-speech (TTS) system to speak new text in the voice of someone it has heard only in a few seconds of reference audio, called the prompt. A single prompt cannot tell the model whether a raised, bright voice comes from the speaker's vocal anatomy or from the fact that they were excited when recorded. Physicists call this situation a parameter degeneracy: two different underlying configurations produce the same observation, so no amount of clean data seen through that one observation can separate them. Today's cloning encoders map each prompt to a single point estimate of "who is speaking". That point sits somewhere arbitrary along the ambiguous direction, and the excitement of the prompt gets baked into the cloned voice for everything it says afterwards.
We propose to borrow the remedy that cosmology uses for exactly this problem. When one measurement leaves a long, narrow valley of equally good answers, you add a second measurement whose valley runs in a different direction, and you combine them by adding their information. Concretely, the speaker encoder will output a Gaussian belief over the speaker embedding instead of a point. The covariance of that Gaussian is shaped to stretch along the directions this particular prompt cannot pin down. Several prompts are then fused by adding their precision matrices, the standard way of adding independent information. The claim is that this calibrated, mergeable uncertainty measurably reduces "identity drift", meaning the change in a cloned voice caused by the emotional state of the prompt. It should also tell the system which extra reference clip would help most. Neither plain averaging of embeddings nor heuristic disentanglement can provide that second capability.
Core Claim
The ambiguity in zero-shot cloning is structured, not random noise. Each prompt leaves a few specific directions in speaker-embedding space unresolved, and those directions depend on the prompt's prosody, meaning its pitch, loudness and rhythm. An angry clip leaves pitch height and vocal effort ambiguous. A whispered or sad clip leaves breathiness and pitch range ambiguous.
If an encoder learns to report these directions as the long axes of a per-prompt covariance, three things follow. First, single-prompt synthesis can avoid committing to the prompt's prosody by sampling or marginalizing along those axes. Second, two prompts with differently oriented ambiguity combine into a much tighter estimate than two prompts with the same orientation. This is the speech analogue of the result in Cosmic Complementarity, which combined CMB and redshift-survey data whose error ellipses cross and cut the uncertainty on the Hubble constant from about 20% to about 3%. Third, the predicted orientation can be checked independently by perturbing the embedding and measuring what changes in the synthesized audio.
Relation to Prior Work
Prior work on the TTS side motivates the gap without formalizing it.
- CrossSpeech++ separates language from speaker using two generators, speaker-mixing normalization and pitch/formant perturbation. Its separation is checked by t-SNE (a 2-D visualization of embeddings) rather than by any measure of how much ambiguity remains.
- DiFlow-TTS factorizes speech into discrete content, prosody and acoustic-detail codec tokens. It concedes that its speaker similarity is not markedly better than baselines.
- ProsodyFlow handles the one-to-many nature of prosody with conditional flow matching, a generator that learns a velocity field carrying noise to data. It is evaluated only on the single-speaker LJSpeech, where speaker–prosody ambiguity cannot arise.
- StyleTTS 2 samples a continuous style latent by diffusion, a generator that learns to denoise random noise into samples, and is our backbone. Its style and speaker information are inferred as point estimates from the reference.
All four treat the conditioning signal as something to be split or sampled. None treats it as an observation with a direction-dependent information content that can be measured and accumulated.
On the degeneracy side, three papers supply methods.
- Mitigating Parameter Degeneracy using Joint Conditional Diffusion Model for WECC Composite Load Model represents a degenerate inverse problem as a distribution over parameters rather than a point. We adopt that stance but use a low-rank Gaussian so beliefs can be fused in closed form.
- Cosmic Complementarity supplies the fusion rule and a constraint. One-sided finite differences give misleadingly tight sensitivity estimates, so our sensitivity probes use two-sided differences or exact autodiff Jacobians.
- Cosmology with cosmic web environments breaks a degeneracy by measuring separately in distinct sub-regimes (voids, walls, filaments, nodes), improving constraints by up to a factor of 5.5. Prompts recorded in different emotional states play that role here.
We are not aware of any paper in the literature we reviewed that applies this machinery to speech synthesis.
Research Program
The encoder. For prompt $k$, the speaker encoder outputs a mean $\mu_k$ and a precision matrix $\Lambda_k = D_k + U_k U_k^\top$ over the speaker embedding $z_s$. Here $D_k$ is diagonal, and $U_k$ has a small number $r$ of columns that encode how confidently the prompt pins down each direction. The covariance $\Lambda_k^{-1}$ is long, i.e. the prompt is uninformative, in the directions this structure leaves weakly constrained.
Fusing prompts. Prompts from the same speaker are treated as independent observations of one shared $z_s$, and their beliefs are combined by
$$\Lambda = \sum_k \Lambda_k, \qquad \mu = \Lambda^{-1} \sum_k \Lambda_k \mu_k .$$
Prosody stays per-utterance and is generated by the backbone's style diffusion from the target text. The identity being synthesized comes only from the fused belief.
Training. Batches contain $K \in \{1,2,3\}$ prompts from one LibriTTS speaker and a held-out target utterance from the same speaker. There are two losses:
- Reconstruction. The backbone's usual reconstruction and adversarial losses on the target, with $z_s$ sampled from the fused belief.
- Calibration. The fused belief from the prompts must assign high Gaussian likelihood to the encoder mean of the held-out utterance.
The calibration loss is what forces the covariance to be honest. If a prompt's emotional state shifts its mean along some direction, that direction must be given low precision, or the held-out clip will be scored as unlikely. No emotion labels are used.
The diagnostic. We probe the trained system independently of its own covariance. For each prompt we compute the two-sided Jacobian of measured output statistics with respect to $z_s$. The statistics are:
- mean and range of the pitch contour (F0, the fundamental frequency),
- energy,
- the embedding from an off-the-shelf speaker-verification model.
We then test whether the encoder's low-precision directions are the ones that move the output's prosodic statistics. This checks that the covariance captures the physical ambiguity and is not an arbitrary regularizer.
Prompt selection. Given a speaker's pool of candidate clips, the system greedily picks the next clip whose precision most increases the smallest eigenvalue of the fused $\Lambda$. This is a label-free version of choosing an orthogonal second probe.
Baselines
Do-nothing reference. This is the unmodified StyleTTS 2 backbone conditioned on a single prompt. It is trained on the same 245-hour LibriTTS setting the paper used for zero-shot adaptation, with identical steps and hyperparameters. It defines the identity drift the field currently accepts.
Strongest existing system. This is also StyleTTS 2, the strongest zero-shot system in the file. The file gives no speaker-similarity figure on LibriTTS. On VCTK it reports a naturalness CMOS (comparative mean opinion score) of −0.02 relative to ground truth and a speaker-similarity CMOS of +0.30. On LJSpeech it reports a CMOS of +0.28 against ground truth.
For ESD, the emotional dataset we evaluate on, there is no reported speaker-similarity or drift result on Lacuna. The only ESD figure is a WER of 4.54 for Emo-DPO, a model trained on ESD for emotion control. We will therefore re-measure StyleTTS 2 ourselves on every metric under identical evaluation.
To beat it, our system must do both of the following:
- Significantly reduce identity drift on ESD with a single prompt, without a naturalness CMOS loss larger than 0.1.
- With two prompts, reduce drift beyond what StyleTTS 2 achieves when it is given those same two prompts.
Factorized-codec baseline. DiFlow-TTS is the main representative of disentanglement through factorized discrete tokens. It is retrained on the same LibriTTS subset. It reports a real-time factor of 0.03 but no speaker-similarity number found on Lacuna.
Ablations. Three ablations isolate the key ingredient:
- Isotropic. The same encoder with $U_k = 0$, so uncertainty has no direction.
- Mean-only. Multi-prompt training with the same compute and plain averaging of means.
- Uncalibrated. Our model with the calibration loss removed.
All systems share data, step budget and a fixed hyperparameter search of equal size.
Why not just…?
Why not just average the embeddings of several prompts? This is the most serious threat. With randomly chosen prompts of mixed emotion, averaging may recover most of the benefit. The mean-only ablation tests this directly. Our prediction is specific: averaging and precision fusion should match when prompts are emotionally similar, and differ when one prompt is extreme, because precision weighting discounts that prompt only along its ambiguous directions. If averaging matches fusion in the extreme-prompt condition too, then the covariance contributes nothing beyond selection, and we will report the work as a measurement study.
Why not just train an emotion-invariant speaker encoder with adversarial gradient reversal or a bottleneck? On ESD, emotion labels make this feasible, so we include a gradient-reversal encoder trained against ESD emotion labels as an oracle-label comparison. The evidence against it as a general solution has two parts. CrossSpeech++ needed a dual-generator architecture because simpler separation let features bleed back together in deeper layers. And such a model only removes the factors someone labeled. Our encoder is trained without labels on LibriTTS, so the decisive comparison is drift on ESD for a model that never saw emotion labels.
Why not just scale data, or rely on a factorized codec? Scale does not remove a many-to-one projection. More data through the same single-prompt lens samples the same ambiguous valley more densely. Data efficiency is already set by StyleTTS 2 outperforming VALL-E with 245 versus 60,000 hours. The codec route is tested by the DiFlow-TTS baseline, whose authors already concede weak speaker similarity.
Proposed Experiments
Data. Training uses LibriTTS (LibriTTS). Zero-shot evaluation uses ESD's speakers, held out from training, and VCTK.
Main testbed. ESD records each speaker reading the same sentences in angry, happy, sad, surprised and neutral styles, which gives a controlled test.
- Identity drift is measured by prompting with one emotion of speaker A and synthesizing neutral new text. Drift is the spread, across the five prompt emotions, of the speaker-verification embedding of the outputs, normalized by between-speaker spread.
- Prosody leakage is the within-speaker correlation between the prompt's F0 statistics and the output's F0 statistics.
Quality metrics. Quality is tracked with:
- WER;
- TTScore-int and TTScore-pro (TTScore), which reports an LCC of 0.53 with human scores on SOMOS, versus 0.32 for WER;
- UTMOS;
- MOS and CMOS listening tests for naturalness and speaker similarity, the latter referenced to the speaker's neutral recordings.
Decisive checks. Three results would decide the question.
- Orientation. The angle between a prompt's low-precision directions and its measured Jacobian directions for prosodic statistics should be far below chance. Its alignment should also predict per-prompt drift for the baseline.
- Complementarity. Fusing two prompts of different emotion (for example angry plus sad) should reduce drift more than fusing two prompts of the same emotion. This is the crossing-ellipse effect, and it should exceed the gain from averaging.
- Selection. Greedy precision-based prompt selection should beat random selection of the same number of clips on drift.
A failure of check 1 would mean the encoder is not representing the degeneracy at all, and the proposal's core claim would be false.
References
- Cosmic Complementarity: Joint Parameter Estimation from Cosmic Microwave Background Experiments and Redshift Surveys
- CrossSpeech++: Cross-Lingual Speech Synthesis With Decoupled Language and Speaker Generation
- DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Factorized Discrete Flow Matching
- ProsodyFlow: High-fidelity Text-to-Speech through Conditional Flow Matching and Prosody Modeling with Large Speech Language Models
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
- Diffusion Model-based Parameter Estimation in Dynamic Power Systems
- Cosmology with cosmic web environments
- Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
- Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokens