Text-Free Non-Parallel Many-To-Many Voice Conversion Using Normalising Flow

Thomas Merritt, Abdelhamid Ezzerg, Piotr Bilinski, Magdalena Proszewska, Kamil Pokora, Roberto Barra-Chicote, Daniel KorzekwaICASSP 2022 · DOI

TL;DR

This paper introduces a novel approach to voice conversion (VC) using normalizing flows, which preserve speech information better than traditional methods. Their text-free VC model achieves state-of-the-art results, demonstrating that explicit text information isn't always necessary for high-quality voice transformation.

What Problem Does This Paper Address?

Voice conversion (VC) aims to change the speaker's voice in an audio recording while preserving the content and prosody. Traditional non-parallel VC methods often rely on "lossy" representations of speech, meaning some information is discarded to achieve the conversion. This makes it challenging to ensure that only speaker identity is altered, and not other crucial aspects like the spoken text or emotional tone. A particular challenge is "text-free" VC, where the model must perform conversion without knowing the actual text being spoken, which is useful for applications like real-time voice filtering. This paper tackles the problem of achieving high-quality, text-free VC by exploring information-preserving techniques, specifically normalizing flows.

Methods

The core of this work is the application of normalizing flows to voice conversion. Normalizing flows are a class of generative models that learn an invertible mapping between data and a latent space. This invertibility is key because it means the latent representation is "lossless," preserving all information from the original speech.

The authors propose two main variants of their flow-based VC approach:

  • Text-conditioned VC (fixed prior): This is similar to existing text-to-speech (TTS) models that use normalizing flows. The model takes mel-spectrograms, phoneme conditioning, speaker embeddings, and prosody information (f0 and voiced/unvoiced flags) to encode speech into a latent vector. For conversion, it uses the target speaker's embedding.
  • Text-free VC (context-dependent prior): This is the more novel aspect. Here, the model does not receive explicit text information during inference. Instead, it learns to represent phonetic context within the latent vector. This is achieved by using a "phoneme-dependent prior" distribution. The authors explore two ways to train this text-free model:
  • Pre-trained phoneme prior: A separate TTS model is used to pre-train a phoneme encoder, and its output is used as the prior for the VC model.
  • Jointly-trained phoneme prior: The phoneme encoder and the flow-based speech encoder are trained together.

The authors also highlight that they use pre-trained phoneme alignments, simplifying the training process compared to methods that learn alignments during training.

Datasets

  • Training Data: Approximately 597,000 utterances from 3173 speakers across six English accents (US, GB, Canadian, Welsh, Australian, Indian) were used for training. The data varied in recording quality and speaker data quantity.
  • Evaluation Data: Six speakers (US male, US female, Australian female, Indian male, Canadian female, Welsh male) were selected from the training data for evaluation. They created 30 conversion combinations between these speakers, evaluating 20 utterances per combination, totaling 600 evaluation screens. Accent similarity evaluations were performed on 560 screens.

Key Insights and Ideas

  • Information Preservation is Key: Normalizing flows offer a significant advantage by preserving speech information, which is crucial for high-quality VC, especially when disentangling speaker identity from other attributes.
  • Text-Free VC is Viable and Effective: The paper demonstrates that high-quality VC can be achieved without explicit text information at inference time, opening up new possibilities for real-time applications.
  • Pre-trained Priors Outperform Joint Training: For text-free VC, using a pre-trained phoneme prior yielded better results than jointly training the prior with the flow model. This suggests that a well-defined prior is beneficial and that joint training might require more sophisticated schedules.
  • Flows Disentangle Speaker Identity from Accent: The results show that the flow-based approach successfully separates speaker identity from accent, meaning the target speaker's accent doesn't overwrite the source speaker's original accent.

Key Results

  • Word Error Rate (WER): The proposed text-free VC approaches (flow text-conditioned, flow text-free pre-train, flow text-free joint-train) achieved comparable WERs to the text-conditioned baseline (CopyCat), significantly outperforming AutoVC (28.97% WER vs. ~12-13% WER). This indicates good intelligibility.
  • Naturalness and Speaker Similarity (MUSHRA tests):
    • Comparing flow-based approaches: Text-conditioned and text-free pre-trained models showed no statistically significant difference in naturalness or speaker similarity, indicating text-free VC doesn't degrade quality within the flow framework. Jointly trained text-free VC performed slightly worse.
  • Comparing to baselines: The flow text-free pre-train model outperformed AutoVC in naturalness and accent similarity. It was competitive with CopyCat in speaker similarity and accent similarity but slightly behind in naturalness, with listeners perceiving minor audio artifacts.

Limitations and Future Work

  • Audio Artifacts: While intelligibility is maintained, listeners perceived minor audio artifacts in the flow-based models compared to CopyCat, leading to a slight preference for CopyCat in naturalness. The authors suggest this is due to the flow decoder lacking recurrent connections for cross-frame awareness.
  • Text-Conditioned vs. Text-Free Gap: While text-free VC with flows is strong, it hasn't quite reached the state-of-the-art of text-conditioned models. This gap is hypothesized to be due to the lack of recurrent connections in the flow decoder.
  • Future Work:
    • Investigating auto-regressive flows to improve cross-frame awareness and reduce artifacts.
  • Exploring improved training schedules for joint-training of priors.
  • Extending the framework to control other speech attributes like accent and prosody independently.

Summary updated

BibTeX
@inproceedings{merritt2022text,
  author = {Thomas Merritt and Abdelhamid Ezzerg and Piotr Bilinski and Magdalena Proszewska and Kamil Pokora and Roberto Barra-Chicote and Daniel Korzekwa},
  title = {{Text-Free Non-Parallel Many-To-Many Voice Conversion Using Normalising Flow}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746368},
}
Figure