Abstract
This paper presents a novel approach to unsupervised voice conversion that improves the naturalness of synthesized speech by enforcing Time-Frequency (TF) consistency. By incorporating a mathematical constraint into a Generative Adversarial Network (GAN) framework, the proposed method ensures that generated magnitude spectrograms correspond to physically realizable audio signals. Experimental results on the LibriSpeech corpus demonstrate that this consistency loss leads to higher quality audio reconstruction and more accurate domain translation compared to baseline unsupervised models.
Introduction
Speech-to-speech conversion, such as male-to-female voice translation, often relies on magnitude spectrograms as an intermediate representation. While computationally efficient for neural networks, magnitude spectrograms do not always represent valid audio signals. When a generated spectrogram is "inconsistent"—meaning it lacks a corresponding phase that satisfies mathematical constraints—reconstruction algorithms like Griffin-Lim produce audible artifacts, including metallic echoes and robotic distortions. Most existing unsupervised models fail to account for this requirement during training, leading to a fundamental gap between generated features and high-quality audio output.
Methodology
The proposed system utilizes the Unsupervised Image-to-image Translation (UNIT) framework. UNIT employs a shared latent space to translate features between two domains (e.g., Male and Female) without requiring parallel training data.
The core innovation is the introduction of a Time-Frequency (TF) Consistency Loss. This loss is based on the principle that a valid log-magnitude spectrogram must maintain a specific relationship between its partial derivatives across time and frequency. By integrating an empirical measure of this consistency as an adversarial loss term, the generator is penalized for producing mathematically invalid spectrograms. This constraint forces the model to learn the structural properties of real human speech rather than merely approximating the visual shape of a spectrogram.
Experimental Setup
The model was evaluated using the LibriSpeech corpus, specifically the train-clean-100 and test-clean-100 subsets. The training set consisted of approximately 48,000 four-second segments for each gender domain. For evaluation, a held-out set of 40 speakers (20 male, 20 female) was used. Audio signals were preprocessed into TF representations using a Short-Time Fourier Transform (STFT) with a 512-frame size and 75% overlap. The researchers found that training on log-magnitude spectrograms significantly outperformed linear magnitude by compressing noise and emphasizing critical low-frequency components.
Key Results
The proposed model, C-UNST (Consistent Unsupervised Speech Translation), was compared against a baseline UNST model lacking the consistency constraint.
Quantitative evaluation using the Fréchet Inception Distance (FID) showed a marked improvement; for Female-to-Male conversion, the FID score dropped from 76.90 to 70.11. Qualitative Mean Opinion Scores (MOS) also favored the C-UNST model in terms of both speech quality and "domainness" (the degree to which the voice matched the target gender). Furthermore, a pre-trained gender classifier achieved up to 92.8% accuracy in identifying the target gender of converted speech, confirming the effectiveness of the translation.
Conclusion and Future Work
Enforcing TF consistency during GAN training significantly enhances the quality of unsupervised voice conversion. While the current approach improves the performance of the Griffin-Lim algorithm, the model still relies on phase approximation. Future research will explore the integration of neural vocoders, such as HiFi-GAN, to eliminate the need for iterative phase reconstruction. Additionally, extending the framework from domain-level conversion to individual speaker identity preservation remains a key area for development. Finally, addressing the high computational cost of training GANs on high-resolution spectrograms will be necessary for broader practical application.
Summary updated