FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion

Jingyi Li, Weiping Tu, Li XiaoICASSP 2022 · OpenReview

TL;DR

This paper introduces FreeVC, a novel text-free, one-shot voice conversion system that achieves high-quality speech synthesis without requiring text transcriptions. By leveraging self-supervised learning and a unique spectrogram resizing data augmentation technique, FreeVC effectively disentangles speaker identity from linguistic content, outperforming existing methods in both naturalness and speaker similarity.

What Problem Does This Paper Address?

Voice conversion (VC) aims to transform speech from a source speaker to a target speaker's voice while preserving the linguistic content. Traditional VC methods often fall into two categories: text-based approaches that rely on accurate transcriptions (which are costly to obtain and can limit performance) and text-free approaches that struggle with speaker information leakage into the extracted content representation. Furthermore, many VC systems use a two-stage pipeline (conversion model + vocoder) which can lead to quality degradation due to a mismatch between the predicted acoustic features and the vocoder's training data. This paper tackles the challenge of achieving high-quality, text-free, one-shot VC by proposing a method that disentangles content and speaker information effectively without needing text annotations and by utilizing an end-to-end framework to mitigate feature mismatch issues.

Methods

FreeVC builds upon the VITS (Variational Inference with Adversarial Learning for End-to-End Text-to-Speech) framework, known for its high-quality waveform reconstruction. The key innovations lie in how content information is extracted and how speaker information is handled:

  • Text-Free Content Extraction: Instead of relying on text-based features like phoneme posteriorgrams (PPGs), FreeVC uses WavLM, a powerful self-supervised speech representation model, to extract features from raw audio. A "bottleneck extractor" is then applied to these WavLM features. This bottleneck imposes a constraint, forcing the model to learn a lower-dimensional representation that ideally discards speaker-specific information, thus isolating the linguistic content.
  • Spectrogram-Resize (SR) Data Augmentation: To further improve the purity of the extracted content information and prevent speaker leakage, the authors introduce a novel data augmentation technique. This involves vertically resizing mel-spectrograms of the source audio. By distorting the frequency bins (e.g., stretching or compressing them), the model is encouraged to focus on content that remains consistent across these distortions, while speaker-specific cues are altered. This augmentation helps the model learn to disentangle content from speaker identity more effectively.
  • One-Shot Speaker Encoding: For one-shot VC (where only a single utterance from the target speaker is available), FreeVC employs a speaker encoder. This encoder extracts a speaker embedding from the target utterance, which is then used to condition the conversion process. The paper explores both pre-trained and jointly trained speaker encoders.
  • End-to-End Framework: By adopting VITS's end-to-end architecture, FreeVC integrates the content extraction, speaker encoding, and waveform generation into a single model. This helps reduce the feature mismatch problem often encountered in multi-stage systems.

Datasets

  • Training: The VCTK corpus was used for training the FreeVC model.
  • Evaluation: Experiments were conducted on the VCTK and LibriTTS datasets. For evaluation, the authors used a combination of seen speakers (from VCTK) and unseen speakers (from LibriTTS) to test performance across different scenarios (seen-to-seen, unseen-to-seen, unseen-to-unseen).

Key Insights and Ideas

  • WavLM's Power for Content Extraction: The success of using WavLM's self-supervised features highlights the potential of these models for downstream tasks like VC, offering a robust alternative to text-based features.
  • Spectrogram Resizing as a Clever Disentanglement Tool: The SR-based data augmentation is a particularly elegant solution. By manipulating the frequency dimension of spectrograms, it effectively "confuses" the speaker encoder without corrupting the core linguistic information, forcing the content encoder to become more robust.
  • Pre-trained Speaker Encoders are Beneficial but Not Essential: While a pre-trained speaker encoder (as used in FreeVC) shows slight advantages, a jointly trained, simpler speaker encoder (FreeVC-s) performs comparably, suggesting that the core disentanglement mechanism is the primary driver of performance.
  • Robustness to Source Quality: FreeVC demonstrates remarkable robustness to variations in source speech quality (e.g., low recording quality, unclear pronunciation), a significant improvement over many existing methods.

Key Results

  • Superior Subjective Performance: FreeVC achieved the highest MOS (Mean Opinion Score) and SMOS (Similarity Mean Opinion Score) across all tested scenarios (seen-to-seen, unseen-to-seen, unseen-to-unseen), outperforming strong baselines like VQMIVC, BNE-PPG-VC, and YourTTS. For instance, in the seen-to-seen scenario, FreeVC achieved a MOS of 3.99, compared to YourTTS's 3.46.
  • Improved Objective Metrics: FreeVC demonstrated significantly lower Word Error Rate (WER) and Character Error Rate (CER) than all baselines, indicating better preservation of linguistic content. It also achieved higher F0-PCC (Pearson correlation coefficient for F0 variation), signifying better prosody preservation.
  • Impact of SR Augmentation: FreeVC (with SR) consistently outperformed FreeVC (w/o SR), underscoring the effectiveness of the proposed data augmentation technique in improving speech naturalness and speaker similarity.

Limitations and Future Work

  • Speaker Adaptation for Unseen Targets: While FreeVC performs well with unseen speakers, the authors suggest that future work could focus on speaker adaptation methods to further improve similarity for unseen target speakers, especially when limited target data is available.
  • Potential for Further Disentanglement: Although the SR augmentation is effective, there might be room for exploring even more sophisticated methods to ensure complete disentanglement of speaker identity from content.

Summary updated

Abstract

Voice conversion (VC) can be achieved by first extracting source content information and target speaker information, and then reconstructing waveform with these information. However, current approaches normally either extract dirty content information with speaker information leaked in, or demand a large amount of annotated data for training. Besides, the quality of reconstructed waveform can be degraded by the mismatch between conversion model and vocoder. In this paper, we adopt the end-to-end framework of VITS for high-quality waveform reconstruction, and propose strategies for clean content information extraction without text annotation. We disentangle content information by imposing an information bottleneck to WavLM features, and propose the spectrogram-resize based data augmentation to improve the purity of extracted content information. Experimental results show that the proposed method outperforms the latest VC models trained with annotated data and has greater robustness.
BibTeX
@inproceedings{li2022freevc,
  author = {Jingyi Li and Weiping Tu and Li Xiao},
  title = {{FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=3v_EqpdtleB},
}
Figure