Self-supervised speech models are networks trained on raw unlabeled audio to predict masked parts of their own input, such as HuBERT and WavLM. They produce frame-level features that everyone uses as "content" for voice conversion, but those features still carry a great deal of speaker identity. The cleanest fix found so far is to define content as the average feature that many different speakers produce when saying the same sentence, and to train a network to predict that average. This works well. It also needs perfectly time-aligned parallel recordings, which exist only when a text-to-speech system is forced to synthesize them. We propose to compute that average without any parallel data. For each rough sound class, we take the per-speaker feature distributions of hundreds of real, unpaired speakers and compute their optimal-transport barycenter, meaning the distribution that is on average closest to all of them. Each speaker's frames are then moved onto that shared distribution in closed form.
This matters for three reasons. First, it turns an idea that currently depends on synthetic parallel data into one that can use any real, diverse, in-the-wild corpus. Second, it makes the number and variety of speakers behind the "average" into a knob we can measure directly. Earlier work on speaker encoders found that speaker diversity matters more than recording cleanliness, and we can test whether the same holds here. Third, the output stays continuous, so it avoids the phonetic hallucinations that come from compressing speech into discrete tokens.
Core Claim
A speaker-invariant content representation comparable to the one learned from synthetic parallel data can be built from unpaired real speech. The method estimates, for every coarse sound unit, a Wasserstein barycenter of per-speaker Gaussian feature distributions, and maps each frame to that barycenter. The mapping is a soft mixture of per-unit closed-form transport maps.
The key ingredient is conditioning on sound units. A single global transport map per speaker is essentially per-speaker feature standardization, and it cannot represent the fact that speaker differences depend on the sound: formant shifts differ between vowels and fricatives. A per-unit map can. We further claim that the representation gets better as the number of speakers in the pool grows, with total hours held fixed. That is the diversity prediction borrowed from Ye Jia, who found 18,000 noisy speakers beat 1,000 clean ones for zero-shot speaker encoding.
How the method builds on and departs from prior work
Two papers supply the pieces. AVENet motivates the target. It generates 15,000 frame-aligned parallel sets across 40 speakers by freezing a text-to-speech duration predictor, then trains a Conformer to map one speaker's HuBERT, WavLM or Whisper features to the multi-speaker average. On AISHELL3 this cut WavLM character error rate from 9.98% to 6.52% and raised naturalness mean opinion score from 4.03 to 4.22. The paper cannot compute its average from real speech, because real speakers never align frame by frame. We keep AVENet's definition of content and replace the parallel-data estimator.
Training-Free Voice Conversion with Factorized Optimal Transport (MKL-VC) supplies the transport machinery. It observes that WavLM-Large features, sorted by variance, are nearly Gaussian within small subspaces of dimension $K=2$, and that only about the first 100 dimensions carry much variance. It then applies the closed-form Monge-Kantorovich linear map from one source speaker to one target speaker. We differ from it in two ways. We transport many speakers to a common barycenter rather than one speaker to another, and we condition the map on sound units rather than using one map per subspace for all frames.
Discrete-token methods constrain the design. Accent normalization with self-supervised discrete tokens and Unit-DSR remove speaker information by k-means quantization of HuBERT features. The first reports post-conversion word error rate above that of the source and misses Hindi retroflex contrasts. We therefore use units only as soft conditioning for choosing a transport map, never as the output. Finally, Gody and Harwath show that unsupervised diversity measures computed on HuBERT units can pick more useful data than random selection. That result motivates choosing barycenter pools by diversity rather than by volume.
Method
We extract WavLM-Large features from a fixed layer. Following MKL-VC, we sort dimensions by variance and split them into 2-D subspaces. We cluster frames into $U$ coarse units with k-means on a middle layer, which the layer-analysis literature identifies as the most phonetic (Glass et al.; Tomashenko et al.). Units are assigned softly with posterior $p(k\mid x)$ at a tuned temperature.
For each speaker $s$, unit $k$ and subspace, we fit a Gaussian with mean $m_{s,k}$ and covariance $\Sigma_{s,k}$. When a speaker has few frames of a unit, we shrink these statistics toward that speaker's global statistics, so short references still work. The barycenter of the $S$ speakers for unit $k$ has mean $\bar m_k$ equal to the average of the $m_{s,k}$. Its covariance $\bar\Sigma_k$ is the fixed point of
$$\bar\Sigma_k = \frac{1}{S}\sum_{s=1}^{S}\left(\bar\Sigma_k^{1/2}\,\Sigma_{s,k}\,\bar\Sigma_k^{1/2}\right)^{1/2},$$
which is the standard characterization of the Gaussian Wasserstein barycenter and converges in a few iterations in 2-D.
A frame $x$ from speaker $s$ is normalized as
$$\tilde x = \sum_{k=1}^{U} p(k\mid x)\left[\bar m_k + A_{s,k}(x - m_{s,k})\right],$$
where $A_{s,k}$ is the closed-form linear map carrying the Gaussian of speaker $s$ for unit $k$ onto the barycenter Gaussian, which is the same form MKL-VC uses. Everything is estimated from unpaired audio with no labels.
For voice conversion, $\tilde x$ goes to a vocoder conditioned on a target speaker embedding. There is also a training-free variant that composes the inverse barycenter map of the target speaker. We use a HiFi-GAN vocoder trained on WavLM features, as in MKL-VC.
Research Program
The work has three parts.
The first part builds the estimator and characterizes the representation directly, without a vocoder. We train linear probes to measure speaker leakage, reported as speaker-verification equal error rate and speaker-ID accuracy on held-out speakers. We train separate probes to measure phonetic content, reported as frame phone accuracy on LibriSpeech alignments.
The second part is the diversity study. We build barycenter pools from LibriSpeech and from in-the-wild corpora, Emilia (He et al.) and GigaSpeech 2 (Yang et al.). The speaker count varies from 20 to several thousand while total hours stay fixed, and noise or acoustic variety varies separately. The question is whether leakage keeps falling with speaker count, and whether heterogeneous acoustics help, as Zhang et al. found for enhancement, or hurt through domain mismatch, as Le and Mower Provost warn for merged clinical sets.
The third part plugs the representation into voice conversion on LibriSpeech and AISHELL3 for direct comparison with MKL-VC and AVENet. It also stress-tests on L2-ARCTIC (L2-ARCTIC), where accent-specific sounds test whether soft unit conditioning keeps contrasts that hard tokens lose.
Baselines
Unmodified WavLM features. The do-nothing reference is the raw features from the same layer fed to the same vocoder. On AISHELL3, AVENet reports 9.98% character error rate and naturalness 4.03 for original WavLM features. We rerun this reference ourselves with our vocoder rather than borrowing the number.
Global barycenter mapping. This is the ablation that isolates the key ingredient. It is identical to our method with $U=1$, which amounts to per-speaker standardization of each subspace followed by recoloring to a pooled barycenter. If unit conditioning does not beat it on leakage at matched phone accuracy, the central claim fails.
MKL-VC, the strongest training-free competitor on LibriSpeech. We run it with the authors' $K=2$ setting and the same vocoder and reference lengths of 5–10 seconds. It reports 8.13% word error rate against 32.29% for kNN-VC and 8.48% for supervised FACodec. Beating it means lower or equal word error rate with higher speaker similarity at the same reference length.
AVENet, the strongest method on the representation question. It needs a Mandarin text-to-speech model and synthetic parallel data. We compare on AISHELL3 using its reported 6.52% character error rate, 4.22 naturalness and 4.28 speaker-similarity opinion scores for WavLM. Where we can, we also retrain it under the same vocoder. Matching it within a small margin without parallel data counts as success. Beating it would be a bonus.
Discrete-token baseline. Following Bai et al., we deduplicate k-means units and use a unit vocoder, with the same $U$. This tests whether continuous transport avoids the reported rise in word error rate.
Every method gets the same WavLM layer, vocoder architecture, training hours and a matched hyperparameter search budget. The only exception is AVENet's inherent need for synthetic data.
Why not just…?
Why not just normalize each speaker's features globally, as in mean-variance normalization? This is the most serious threat. The global barycenter ablation is exactly that alternative. Our claim that speaker effects depend on the sound is plausible from the known formant physics, but it is unproven in WavLM space. If the global map matches our leakage and word error rate, we will report that the simpler method suffices and that the barycenter framing adds nothing beyond pooling speakers.
Why not just use HuBERT units with deduplication? Bai et al. report that this route leaves converted speech with higher word error rate than the source and fails on Hindi retroflexes. Unit-DSR degrades on very slow speech. Our output stays continuous, and the L2-ARCTIC per-accent word error rate comparison, especially on Hindi, tests directly whether soft conditioning keeps what quantization discards.
Why not just run MKL-VC to one fixed "average" reference speaker? Mapping everyone to a single real speaker imposes that speaker's idiosyncrasies rather than removing identity, and MKL-VC itself notes it has no multi-speaker aggregation. The diversity sweep settles this, because a one-speaker pool is its endpoint. If leakage does not drop as the pool grows, this alternative is as good.
Why not just synthesize parallel data as AVENet does? That approach ties the method to a good multi-speaker text-to-speech model in the target language and to read, synthetic speech. Our method runs on unscripted in-the-wild corpora. The AISHELL3 comparison shows how much, if anything, is lost by dropping parallel data.
Proposed Experiments and Decisive Checks
The first decisive check is the $U$ sweep from 1 up to 200 at fixed pool size. We expect speaker-verification equal error rate on normalized features to rise with $U$ while phone probe accuracy stays within about one point of raw WavLM. If phone accuracy instead collapses at large $U$, the method is behaving like quantization, and we will choose $U$ at the knee.
The second is the diversity sweep at fixed hours, run on both read and in-the-wild pools. The Jia-style prediction is that the leakage curve is monotone in speaker count. A flat curve would falsify the diversity argument for this representation.
The third is voice conversion. On LibriSpeech we report word error rate, speaker-embedding similarity and naturalness against MKL-VC's 8.13%. On AISHELL3 we report character error rate and opinion scores against AVENet's 6.52% and 4.22. On L2-ARCTIC we report per-accent word error rate against the token baseline, with Hindi as the targeted failure case.
The fourth is a reference-length sweep from 5 seconds to 5 minutes, which checks that shrinkage keeps per-unit statistics usable at the short references where MKL-VC is strong.
One boundary is stated in advance. Like MKL-VC, this is a frame-level method and cannot repair rhythm. Dysarthric speech, where El Hajal et al. show that temporal distortion dominates, is therefore out of scope for the claim.
References
- Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
- AVENet: Disentangling Features by Approximating Average Features for Voice Conversion
- Training-Free Voice Conversion with Factorized Optimal Transport
- Accent Normalization Using Self-Supervised Discrete Tokens with Non-Parallel Data
- UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization
- Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models
- Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
- Privacy Attacks for Automatic Speech Recognition Acoustic Models in A Federated Learning Framework
- Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation
- GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement
- Beyond Performance Plateaus: A Comprehensive Study on Scalability in Speech Enhancement
- Improving Automatic Recognition of Aphasic Speech with AphasiaBank
- L2-ARCTIC: A Non-native English Speech Corpus
- Unsupervised Rhythm and Voice Conversion to Improve ASR on Dysarthric Speech