Unsupervised Cross-Domain Speech-to-Speech Conversion with Time-Frequency Consistency

Mohammad Asif Khan, Fabien Cardinaux, Stefan Uhlich, Marc Ferras, Asja FischerarXiv 2020 · OpenReview

Abstract

This paper presents a novel approach to unsupervised voice conversion that improves the naturalness of synthesized speech by enforcing Time-Frequency (TF) consistency. By incorporating a mathematical constraint into a Generative Adversarial Network (GAN) framework, the proposed method ensures that generated magnitude spectrograms correspond to physically realizable audio signals. Experimental results on the LibriSpeech corpus demonstrate that this consistency loss leads to higher quality audio reconstruction and more accurate domain translation compared to baseline unsupervised models.

Introduction

Speech-to-speech conversion, such as male-to-female voice translation, often relies on magnitude spectrograms as an intermediate representation. While computationally efficient for neural networks, magnitude spectrograms do not always represent valid audio signals. When a generated spectrogram is "inconsistent"—meaning it lacks a corresponding phase that satisfies mathematical constraints—reconstruction algorithms like Griffin-Lim produce audible artifacts, including metallic echoes and robotic distortions. Most existing unsupervised models fail to account for this requirement during training, leading to a fundamental gap between generated features and high-quality audio output.

Methodology

The proposed system utilizes the Unsupervised Image-to-image Translation (UNIT) framework. UNIT employs a shared latent space to translate features between two domains (e.g., Male and Female) without requiring parallel training data.

The core innovation is the introduction of a Time-Frequency (TF) Consistency Loss. This loss is based on the principle that a valid log-magnitude spectrogram must maintain a specific relationship between its partial derivatives across time and frequency. By integrating an empirical measure of this consistency as an adversarial loss term, the generator is penalized for producing mathematically invalid spectrograms. This constraint forces the model to learn the structural properties of real human speech rather than merely approximating the visual shape of a spectrogram.

Experimental Setup

The model was evaluated using the LibriSpeech corpus, specifically the train-clean-100 and test-clean-100 subsets. The training set consisted of approximately 48,000 four-second segments for each gender domain. For evaluation, a held-out set of 40 speakers (20 male, 20 female) was used. Audio signals were preprocessed into TF representations using a Short-Time Fourier Transform (STFT) with a 512-frame size and 75% overlap. The researchers found that training on log-magnitude spectrograms significantly outperformed linear magnitude by compressing noise and emphasizing critical low-frequency components.

Key Results

The proposed model, C-UNST (Consistent Unsupervised Speech Translation), was compared against a baseline UNST model lacking the consistency constraint.

Spectrograms of source and converted speech across male and female domains.

Quantitative evaluation using the Fréchet Inception Distance (FID) showed a marked improvement; for Female-to-Male conversion, the FID score dropped from 76.90 to 70.11. Qualitative Mean Opinion Scores (MOS) also favored the C-UNST model in terms of both speech quality and "domainness" (the degree to which the voice matched the target gender). Furthermore, a pre-trained gender classifier achieved up to 92.8% accuracy in identifying the target gender of converted speech, confirming the effectiveness of the translation.

Conclusion and Future Work

Enforcing TF consistency during GAN training significantly enhances the quality of unsupervised voice conversion. While the current approach improves the performance of the Griffin-Lim algorithm, the model still relies on phase approximation. Future research will explore the integration of neural vocoders, such as HiFi-GAN, to eliminate the need for iterative phase reconstruction. Additionally, extending the framework from domain-level conversion to individual speaker identity preservation remains a key area for development. Finally, addressing the high computational cost of training GANs on high-resolution spectrograms will be necessary for broader practical application.

Summary updated

Abstract

In recent years generative adversarial network (GAN) based models have been successfully applied for unsupervised speech-to-speech conversion.The rich compact harmonic view of the magnitude spectrogram is considered a suitable choice for training these models with audio data. To reconstruct the speech signal first a magnitude spectrogram is generated by the neural network, which is then utilized by methods like the Griffin-Lim algorithm to reconstruct a phase spectrogram. This procedure bears the problem that the generated magnitude spectrogram may not be consistent, which is required for finding a phase such that the full spectrogram has a natural-sounding speech waveform. In this work, we approach this problem by proposing a condition encouraging spectrogram consistency during the adversarial training procedure. We demonstrate our approach on the task of translating the voice of a male speaker to that of a female speaker, and vice versa. Our experimental results on the Librispeech corpus show that the model trained with the TF consistency provides a perceptually better quality of speech-to-speech conversion.
BibTeX
@misc{khan2020unsupervised,
  author = {Mohammad Asif Khan and Fabien Cardinaux and Stefan Uhlich and Marc Ferras and Asja Fischer},
  title = {{Unsupervised Cross-Domain Speech-to-Speech Conversion with Time-Frequency Consistency}},
  year = {2020},
  eprint = {2005.07810},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.07810},
}
Figure