TL;DR
S3PRL-VC is a new open-source framework that proves self-supervised speech representations (S3Rs) can effectively replace expensive, human-labeled data for high-quality voice conversion. By benchmarking over a dozen S3R models, the authors demonstrate that these unsupervised features achieve speaker similarity comparable to top-tier supervised systems while setting a new state-of-the-art for "any-to-any" (zero-shot) voice conversion.
What Problem Does This Paper Address?
Voice conversion (VC) aims to change how someone speaks (their identity) without changing what they say (the content). The gold standard approach is "recognition-synthesis," where a model first extracts linguistic content and then a second model "re-synthesizes" it in the target voice.
Historically, this required supervised models trained on massive amounts of transcribed speech to extract features like text or Phonetic Posteriorgrams (PPGs). This is a major bottleneck: labeled data is expensive to produce and often unavailable for low-resource languages or complex cross-lingual scenarios. S3PRL-VC investigates whether we can ditch the labels and use self-supervised representations—features learned from raw, unlabeled audio—to achieve the same results.
Methods
The authors built S3PRL-VC as an extension of the popular S3PRL toolkit. The framework follows a modular pipeline:
- The Recognizer (Upstream): Extracts content representations ($H$) from source speech ($X$). The paper benchmarks 13 different S3R models (like wav2vec 2.0, HuBERT, and APC) against traditional mel-spectrograms and supervised PPGs.
- The Synthesizer (Downstream): Converts the content features into target acoustic features. The authors tested three architectures:
- Simple: A basic feed-forward and LSTM network.
- Simple-AR: Adds an autoregressive (AR) loop to improve speech flow.
- Taco2-AR: A complex architecture based on Tacotron 2 (minus the attention module).
- Waveform Synthesis: A HiFi-GAN vocoder turns the acoustic features back into audible speech.
For "any-to-any" (A2A) conversion, where the target speaker is unknown during training, they integrated a pretrained d-vector speaker encoder to provide the necessary identity "embedding" to the synthesizer.
Datasets
- VCC2020: Used for benchmarking intra-lingual (English-to-English) and cross-lingual (English to Finnish, German, or Mandarin) any-to-one conversion.
- VCTK: A multi-speaker English dataset used to train the any-to-any (A2A) models.
- LibriSpeech & VoxCeleb: Used to pretrain the upstream S3R models and the d-vector speaker encoder.
Key Insights and Ideas
- The AR Advantage: Adding an autoregressive loop to the synthesizer (the
ARmodels in Figure 2) dramatically improves speaker similarity. However, there is a trade-off: it can slightly increase the Word Error Rate (WER), meaning the speech is more "target-like" but occasionally less intelligible. - Cross-Lingual Transfer: S3R models trained only on English audio generalize surprisingly well to other languages. This suggests they are learning fundamental phonetic structures rather than just English-specific vocabulary.
- Disentanglement is the Holy Grail: The best content representation ($H$) should contain zero information about the original speaker. The authors found that vq-wav2vec and HuBERT are particularly good at this "disentanglement," leading to higher similarity scores because the synthesizer isn't "confused" by the source speaker's traits.
- MCD as a Proxy: The study found that Mel Cepstrum Distortion (MCD) is the objective metric that correlates most strongly with how humans perceive both naturalness and speaker similarity in VC tasks.
Key Results
The benchmarks show that S3R-based systems are now competitive with the best supervised systems in the world.
- Any-to-One (A2O): In the VCC2020 benchmark, the S3R-based
vq-wav2vecmodel achieved speaker similarity scores on par with top-performing systems that used supervised PPGs. - Any-to-Any (A2A): The framework achieved state-of-the-art performance for S3R-based zero-shot conversion, outperforming previous benchmarks like S2VC in both naturalness and similarity.
- Top Performer:
vq-wav2vecemerged as the overall winner among S3Rs, providing the best balance of intelligibility (low WER) and similarity (high ASV accept rate).
Limitations and Future Work
While similarity is high, a "naturalness gap" still remains between S3R-based models and the absolute top-tier supervised systems from VCC2020 (which often use massive internal datasets and complex vocoder fine-tuning).
The authors suggest two main paths forward:
- Better Downstream Models: Moving beyond fixed d-vectors to more sophisticated speaker encoders for A2A tasks.
- VC as a Probe: Using voice conversion performance as a "probing task" to design even better self-supervised representations that are specifically optimized for speech generation rather than just recognition.
Summary updated