Abstract
This paper introduces S3PRL-VC, an open-source voice conversion (VC) framework based on the S3PRL toolkit. In the context of recognition-synthesis VC, self-supervised speech representation (S3R) is valuable in its potential to replace the expensive supervised representation adopted by state-of-the-art VC systems. Moreover, we claim that VC is a good probing task for S3R analysis. In this work, we provide a series of in-depth analyses by benchmarking on the two tasks in VCC2020, namely intra-/cross-lingual any-to-one (A2O) VC, as well as an any-to-any (A2A) setting. We also provide comparisons between not only different S3Rs but also top systems in VCC2020 with supervised representations. Systematic objective and subjective evaluation were conducted, and we show that S3R is comparable with VCC2020 top systems in the A2O setting in terms of similarity, and achieves state-of-the-art in S3R-based A2A VC. We believe the extensive analysis, as well as the toolkit itself, contribute to not only the S3R community but also the VC community.
Introduction
Voice conversion (VC) aims to modify a source speaker's identity to match a target speaker while preserving the original linguistic content. The current gold standard for high-quality VC is the "recognition-synthesis" approach. This paradigm involves a recognition model extracting content-rich representations and a synthesis model re-generating speech in the target voice.
Historically, these systems have relied on supervised models trained on massive amounts of human-labeled data to extract features such as Phonetic Posteriorgrams (PPGs). However, labeled data is a significant bottleneck, particularly for low-resource languages or complex cross-lingual scenarios. S3PRL-VC addresses this by exploring self-supervised speech representations (S3Rs)—features learned from raw, unlabeled audio—as a viable, label-free alternative for content extraction.
Methodology
S3PRL-VC is built as an extension of the popular S3PRL toolkit and utilizes a modular pipeline consisting of three primary components:
- Recognizer (Upstream): This module extracts content representations ($H$) from the source speech ($X$). The framework benchmarks 13 distinct S3R models, including wav2vec 2.0, HuBERT, and APC, comparing them against baseline mel-spectrograms and supervised PPGs.
- Synthesizer (Downstream): This module maps content features to target acoustic features. Three architectures were evaluated: Simple (feed-forward + LSTM), Simple-AR (adding an autoregressive loop), and Taco2-AR (a Tacotron 2-based decoder).
- Waveform Synthesis: A HiFi-GAN vocoder is used to convert the predicted acoustic features into the final audible speech waveform.
For any-to-any (A2A) conversion, where the target speaker is unknown during training, the framework incorporates a pretrained d-vector speaker encoder to provide identity embeddings to the synthesizer.
Experimental Results and Analysis
The benchmarking process utilized the VCC2020 dataset for intra-lingual and cross-lingual any-to-one (A2O) tasks, and the VCTK dataset for any-to-any (A2A) tasks.
Performance of S3Rs
The results demonstrate that S3R-based systems are highly competitive with top-tier supervised systems. Specifically, vq-wav2vec emerged as a leading representation, achieving speaker similarity scores on par with supervised PPG systems in the VCC2020 benchmark. It provided an optimal balance between intelligibility (low Word Error Rate) and speaker similarity (high ASV accept rate).
Key Insights
- Autoregressive (AR) Advantage: Incorporating an AR loop in the synthesizer significantly boosts speaker similarity, making the output more "target-like," though it can sometimes lead to a slight increase in Word Error Rate (WER).
- Disentanglement: Models like
vq-wav2vecandHuBERTexcel at "disentangling" content from speaker identity. By removing source speaker traits from the representation $H$, the synthesizer can more accurately adopt the target identity. - Cross-Lingual Generalization: S3Rs trained on English audio generalize effectively to other languages (Finnish, German, Mandarin), indicating they capture universal phonetic structures rather than language-specific vocabulary.
- Metric Correlation: Mel Cepstrum Distortion (MCD) was found to be the objective metric most strongly correlated with human perceptions of both naturalness and speaker similarity.
Conclusion
S3PRL-VC demonstrates that self-supervised representations can effectively replace expensive supervised features in voice conversion pipelines. While a "naturalness gap" remains compared to the most complex proprietary systems, S3R-based models achieve state-of-the-art performance in open-source A2A VC. Future work will focus on improving speaker encoders for zero-shot tasks and utilizing VC as a probing tool to refine S3R architectures for generative speech tasks.
Summary updated