TEXT-FREE NON-PARALLEL MANY-TO-MANY VOICE CONVERSION USING NORMALISING FLOWS

Thomas Merritt, Abdelhamid Ezzerg, Piotr Biliński, Magdalena Proszewska, Kamil Pokora, Roberto Barra-Chicote, Daniel KorzekwaarXiv 2022 · DOI · arXiv · OpenAlex

Abstract

Non-parallel voice conversion (VC) is typically achieved using lossy representations of the source speech. However, ensuring only speaker identity information is dropped whilst all other information from the source speech is retained is a large challenge. This is particularly challenging in the scenario where at inference-time we have no knowledge of the text being read, i.e., text-free VC. To mitigate this, we investigate information-preserving VC approaches. Normalising flows have gained attention for text-to-speech synthesis, however have been under-explored for VC. Flows utilize invertible functions to learn the likelihood of the data, thus provide a lossless encoding of speech. We investigate normalising flows for VC in both text-conditioned and text-free scenarios. Furthermore, for text-free VC we compare pre-trained and jointly-learnt priors. Flow-based VC evaluations show no degradation between text-free and text-conditioned VC, resulting in improvements over the state-of-the-art. Also, joint-training of the prior is found to negatively impact text-free VC quality.

Introduction

Voice Conversion (VC) aims to modify the speaker identity of an audio signal while preserving its linguistic content and prosodic style. Traditional non-parallel VC systems often rely on an "information bottleneck" to disentangle speaker identity from content. This approach assumes that by compressing the audio through a narrow latent space, speaker-specific details are discarded while phonetic information is retained. However, this is a difficult trade-off: a bottleneck that is too restrictive leads to robotic, unintelligible speech, while one that is too permissive allows the source speaker's identity to "leak" into the converted output.

This challenge is amplified in text-free VC, where the system lacks access to a transcript during inference. Without text to guide reconstruction, maintaining intelligibility becomes paramount. This paper proposes a shift from lossy compression to a lossless, invertible representation using Normalising Flows.

Methodology

The core of the proposed system is a Normalising Flow, a generative model that learns an invertible mapping between complex mel-spectrograms and a simple latent space (typically a Gaussian distribution). Because the mapping is invertible, the latent representation $z$ is mathematically lossless, preserving all information from the source speech.

Architecture and Conditioning

To achieve voice conversion, the flow is conditioned on several factors: speaker embeddings, pitch (normalized $f_0$), and voiced/unvoiced flags. During training, the model learns to map the mel-spectrogram to a latent space where these conditioned factors are "factored out." At inference, conversion is performed by encoding the source speech into the latent space and then decoding it using the target speaker's embedding.

Architecture of the proposed flow-based VC system, including the speech encoder/decoder and phoneme conditioning modules.

Text-Free Priors

For text-free scenarios, the model must handle phonetic context without explicit text input. The authors explored two prior configurations:

  1. Pre-trained "Expert" Prior: A fixed, pre-trained TTS-based phoneme encoder defines the target distribution for the latent space.
  2. Jointly-trained Prior: The phoneme encoder and flow are trained simultaneously.

The study found that using a fixed, pre-trained expert prior significantly improved convergence and final quality compared to joint training.

Experimental Results

The models were trained on a large-scale dataset of approximately 597,000 utterances from over 3,000 speakers across six regional accents. Evaluation focused on intelligibility (Word Error Rate) and subjective quality (MUSHRA).

Intelligibility and Naturalness

A key finding was that the flow-based models maintained high performance even in text-free settings. The text-free flow model achieved a Word Error Rate (WER) of 12.38%, which is comparable to its text-conditioned counterpart (12.65%) and vastly superior to the state-of-the-art text-free baseline, AutoVC, which suffered from a 28.97% WER.

Comparison of Word Error Rate (WER) across different VC systems, demonstrating the superior intelligibility of flow-based models.

Subjective tests also showed that flow models were rated significantly higher in naturalness than AutoVC. Notably, there was no statistically significant difference in naturalness or speaker similarity between the text-conditioned and text-free flow models when using the pre-trained prior.

Accent Preservation

Unlike traditional bottleneck models that often "wash out" the source speaker's accent, the lossless nature of the flow-based approach allowed for the preservation of the source accent while successfully changing the voice identity. This is a direct benefit of the flow's ability to disentangle rather than discard information.

Conclusion

This research demonstrates that Normalising Flows offer a powerful alternative to bottleneck-based voice conversion. By utilizing lossless representations and effective conditioning, the proposed system achieves high-quality, intelligible voice conversion in both text-conditioned and text-free scenarios. The parity between text-free and text-conditioned performance suggests that the flow's latent space implicitly captures necessary phonetic context. Future work will focus on incorporating auto-regressive components to further reduce minor audio artifacts and extending the framework for finer control over prosodic elements.

Summary updated

Abstract

Non-parallel voice conversion (VC) is typically achieved using lossy representations of the source speech. However, ensuring only speaker identity information is dropped whilst all other information from the source speech is retained is a large challenge. This is particularly challenging in the scenario where at inference-time we have no knowledge of the text being read, i.e., text-free VC. To mitigate this, we investigate information-preserving VC approaches. Normalising flows have gained attention for text-to-speech synthesis, however have been under-explored for VC. Flows utilize invertible functions to learn the likelihood of the data, thus provide a lossless encoding of speech. We investigate normalising flows for VC in both text-conditioned and text-free scenarios. Furthermore, for text-free VC we compare pre-trained and jointly-learnt priors. Flow-based VC evaluations show no degradation between text-free and text-conditioned VC, resulting in improvements over the state-of-the-art. Also, joint-training of the prior is found to negatively impact text-free VC quality.
BibTeX
@misc{merritt2022text,
  author = {Thomas Merritt and Abdelhamid Ezzerg and Piotr Biliński and Magdalena Proszewska and Kamil Pokora and Roberto Barra-Chicote and Daniel Korzekwa},
  title = {{TEXT-FREE NON-PARALLEL MANY-TO-MANY VOICE CONVERSION USING NORMALISING FLOWS}},
  year = {2022},
  doi = {10.17023/hj2z-9m52},
}
Figure