Do Voice Conversion Gains Survive a Fixed Pipeline as Speaker Count Grows?

Voice conversion makes one person's recording sound as if a different person said the same words. The goal of "any-to-any" conversion is to do this for speakers the system never saw in training, using one model rather than one model per speaker pair. The field has two competing recipes. The first strips speaker identity out of a content representation and then re-inserts the target voice. The second skips explicit separation and uses attention to copy small acoustic fragments from the target recording. Each recipe reports wins over the other. Those wins, however, are measured on different subsets of the same corpora, with different vocoders (the neural networks that turn spectrograms back into waveforms), different speech recognizers for word error, and different speaker-verification models for similarity. Other fields have learned that such choices can reorder a leaderboard by themselves. In novel view synthesis, NerfBaselines showed that image downscaling and background color flip method rankings. In recommender systems, BARS showed that claimed gains vanish once baselines are tuned under one pipeline.

This proposal asks one sharp question. When any-to-any methods are trained on the same data and scored by a locked, containerized evaluation pipeline, do their rankings hold, and does the ranking change as the number of training speakers grows from about a hundred to a few thousand? The proposal's distinctive measurement is a variance decomposition. It splits the observed differences in conversion quality into four parts: method, number of training speakers, random seed, and evaluator choice. The evaluator choices are which vocoder, which speech recognizer, which speaker verifier, the sampling rate, and how target reference utterances are chosen. If evaluator choice explains as much variance as method choice, the field's current comparisons are not informative. If it does not, we get the first trustworthy scaling curves for the two recipes.

Core Claim

The claim has two parts, and both can be falsified. First, the published spread between any-to-any methods is comparable in size to the spread that evaluation-pipeline choices alone produce. A concrete example is speaker-verification accuracy for unseen speakers: FragmentVC reports 92.5%, AdaIN-VC 87.1%, and AUTOVC 19.0%. Second, the more interesting question about scaling is not answered by any single paper. Do bottleneck-and-disentangle methods and fragment-retrieval methods gain differently from more training speakers, and does source-speaker leakage shrink with scale for either recipe? Gap 3 of the research file names the tension between the two recipes. No one has tested it under matched conditions, and no one has tested it along the speaker-count axis that "scalability" is supposed to describe.

Positioning Against Prior Work

The motivating evidence comes from three places. The VCC 2018 challenge fixed data for 12 speakers and found that a simple GMM baseline beat most custom systems on naturalness. It also found that the winning team used external pretraining data, which confounds data scale with architecture. The VCC 2020 analysis by Das et al. scored pre-rendered challenge submissions with five automatic assessors. It found that speaker-verification equal error rate correlates with human similarity ratings at about r ≈ 0.90, but that automatic predictions were much less reliable in cross-lingual conditions. SRC4VC showed that a studio-trained any-to-any model degrades badly on smartphone input. That study tested only one architecture.

Our work differs from all three. We retrain every method ourselves under a single data and compute budget, rather than scoring audio that participants produced on their own setups. We cover arbitrary unseen speakers rather than a closed set of pairs. We treat evaluator choice as an experimental factor, rather than fixing one evaluator and trusting it.

The methodological lens comes from A Sober Look at Unsupervised Disentanglement. That study trained over 14,000 models and showed that seeds and hyperparameters dominated disentanglement scores, and that the metrics disagreed with one another. We borrow its design: many seeds, a grid over factors, and attribution of variance. We do not assume its conclusion carries over to speech. Voice conversion methods use strong inductive biases, so architecture may well matter here. Content representations from speech recognizers or from self-supervised models such as WavLM are one such bias. Another is the permutation-invariant speaker encoder of S2CD. Measuring whether architecture matters is part of the question.

The systems under test supply the methods. On the disentanglement side they are W2VC, which uses gradient reversal and phoneme supervision on WavLM features; the self-supervised zero-shot model of Training Robust Zero-Shot VC; S2CD; and GAZEV, which conditions a GAN through adaptive instance normalization. On the retrieval side the system is FragmentVC.

Research Program

We will build one container image per method. Each image wraps that method's training and inference code behind a single interface, so it receives the same waveform lists and returns converted waveforms. All evaluation runs in a separate, frozen evaluation container. That container fixes three things for scoring. First, it fixes resampling, loudness normalization, and silence trimming. Second, it fixes the speech recognizer and the text normalization used for word and character error rates. Third, it fixes the speaker-verification model used both for target similarity and for source leakage. Source leakage is measured as the verifier score between the converted audio and the original source speaker. The evaluation container also ships with a small set of swappable alternatives: two speech recognizers, two speaker verifiers, two sampling rates, and two rules for choosing the target reference utterance (single versus several). These alternatives are what define the evaluator factor.

For every method that predicts mel-spectrograms, we train one shared HiFi-GAN vocoder on each training set. This separates vocoder quality from conversion quality. W2VC's vocoder consumes WavLM features rather than spectrograms, so it cannot share this vocoder. We report it both with its own vocoder and, where adaptation is possible, with a spectrogram head. We flag this as a residual confound rather than hiding it.

Training data scale is varied using speaker subsets of LibriTTS, which has 2,456 speakers. The subsets contain roughly 100, 300, 1,000, and all speakers. We hold total training hours approximately constant across subsets, so that more speakers does not quietly mean more data. A separate run without that constraint recovers the realistic scaling curve. VCTK serves as the in-domain unseen-speaker test, with a fixed public speaker split. We add a degraded-input stress test: SRC4VC smartphone recordings, plus studio VCTK passed through synthetic coloration and noise, which SRC4VC identifies as the main drivers of degradation. Each configuration is trained with three to five seeds, depending on what compute allows.

We then fit a random-effects model to each metric $y$, which can be character error rate, target similarity, source leakage, or predicted naturalness:

$$y = \mu + a_{\text{method}} + b_{\text{speakers}} + (ab)_{\text{method}\times\text{speakers}} + c_{\text{seed}} + d_{\text{evaluator}} + \varepsilon.$$

Here $\mu$ is the grand mean. The terms $a$, $b$, $c$, and $d$ are the contributions of method, training speaker count, seed, and evaluator configuration. The term $(ab)$ captures whether methods scale differently with speaker count. The residual $\varepsilon$ collects per-utterance noise. The quantities we report are the share of variance attributed to each term. We also report Kendall's rank correlation between method orderings produced under different evaluator configurations.

Baselines

The naive reference points come first. The first is copy-synthesis: source audio passed unconverted through the shared vocoder. It gives the leakage ceiling, the similarity floor, and the error floor the vocoder imposes. The second is ground-truth target speech resynthesized by the same vocoder. It gives the upper bound for naturalness and similarity. A method has to sit clearly between these two references to count as converting at all.

The strongest existing disentanglement competitor on content preservation is Training Robust Zero-Shot VC. It reports 6.15% character error rate and naturalness MOS 3.66 on VCTK, against FragmentVC's 3.24, and 4.25% character error rate on LibriTTS-460, compared with 4.15% for human recordings. On reported naturalness the strongest is W2VC, with MOS 4.45 and 1.63% word error rate on VCTK. The strongest retrieval competitor is FragmentVC, at 92.5% unseen-speaker verification accuracy. These numbers come from different protocols, so we do not treat them as targets to beat. We treat them as claims to reproduce.

Each system is retrained on our splits. Each gets the same GPU-hour budget and the same number of hyperparameter trials, spent on a random search over the original papers' documented ranges. Every system is then scored in the frozen evaluation container. "Beating" a system means a difference that exceeds the 95% interval across seeds and is preserved under every evaluator configuration, that is, a rank correlation of 1 for that pair. A difference that appears under only some evaluators is reported as unresolved.

Two ablations isolate the proposal's key ingredient, which is the locked evaluator. In the first, each method is evaluated under its original paper's evaluator choices: its own vocoder, its own speech recognizer, and its own verifier. We then check whether published orderings reappear only under those choices. The second removes seed replication, which tests whether single-run comparisons would have reached the same conclusions.

Why not just…?

Why not just rely on the Voice Conversion Challenges? VCC 2018 and VCC 2020 fix the data but score audio that teams produced on unstandardized setups, sometimes with external pretraining data, as VCC 2018 documents for its winning team. They also cover a closed set of speakers. As a result, they cannot answer how methods scale with training speakers or how they behave on speakers never seen. We use VCC 2020 as a validation resource, not as a substitute.

Why not just use speaker-verification error rate and speech-recognition error, given their strong correlation with human ratings? This is the real threat. If a single fixed verifier and recognizer already rank systems reliably, much of the containerization effort is unnecessary. Our design settles the question directly. If the evaluator term $d$ explains little variance and orderings agree across evaluators, we will say so, and the contribution shrinks to the scaling curves. Two findings make us expect otherwise. Das et al. found the correlations much weaker in cross-lingual conditions, and the Sober Look study found metrics disagreeing across configurations. Neither finding has been tested on any-to-any systems.

Why not just add a strong neural vocoder and a speech-enhancement front end to a simple converter? This could plausibly close the naturalness gap. It is less likely to close the leakage gap: W2VC reports that self-supervised features keep source-speaker traits without explicit purification. SRC4VC also reports that enhancement raised smartphone naturalness only from 2.54 to 2.74. We include this alternative as a system in the comparison: AdaIN-VC with the shared HiFi-GAN and a Miipher-style enhancer. If it matches the specialized methods on both similarity and leakage, that is a headline result rather than an embarrassment.

Why not just report multiple seeds within existing papers? Seeds capture only the term $c$. Evaluator choice $d$ and matched training data can produce orderings that are just as unstable, and only a shared harness can measure them.

Proposed Experiments and Decisive Checks

The first decisive check is reproduction. In the matched VCTK setting, do the published orderings hold? Specifically, does FragmentVC beat AdaIN-VC on unseen-speaker verification, and does the self-supervised zero-shot model beat FragmentVC on naturalness? We measure this with seed intervals and rank correlation across all evaluator configurations.

The second check is the variance decomposition. We pre-register the hypothesis that, for at least one of target similarity or source leakage, the evaluator share of variance is at least as large as the method share. We report the result whichever way it goes.

The third check is the scaling interaction. Does the method-by-speaker-count term show a significant difference between retrieval and disentanglement in how similarity and leakage change from 100 to 2,456 speakers, at constant hours? Retrieval needs only good content queries, so it may saturate early. Disentanglement methods may benefit more from speaker diversity in their speaker encoders.

The fourth check is evaluator validity. We run a modest human listening test on a stratified sample. Where evaluators disagree, the human test identifies which automatic configuration tracks listeners best. Where the VCC 2020 human ratings are available, we also check our evaluator configurations against them.

The final check is robustness on degraded input. We ask whether the matched-condition ranking survives on smartphone and synthetically degraded input. SRC4VC is Japanese, so that portion doubles as a cross-lingual stress test. Given Das et al.'s cross-lingual finding, we expect automatic metrics to be least trustworthy there, and we will report it that way.