Generative Adversarial Networks for Unpaired Voice Transformation on Impaired Speech

Li-Wei Chen, Hung-yi Lee, Yu Tsao

Abstract

This paper presents a novel Generative Adversarial Network (GAN) framework for unpaired voice conversion, specifically targeting the restoration of speech intelligibility for patients with oral-articulatory impairments. Unlike traditional methods requiring parallel datasets, the proposed architecture decouples clear speech generation from impaired-to-normal mapping. By utilizing a three-component system—Generator, Discriminator, and Controller—the model transforms distorted audio into clear speech while preserving the patient's unique vocal identity. Experimental results demonstrate significant improvements in articulation and speaker similarity over baseline unsupervised models like CycleGAN.

Introduction

Patients who have undergone oral surgery, such as the removal of parts of the tongue or jaw, often suffer from severely distorted and unintelligible speech. While Voice Conversion (VC) technology offers a potential solution, it faces significant challenges in clinical settings. Most VC models rely on "parallel data"—recordings of the same phrases spoken by both the patient and a healthy individual—which is difficult and exhausting to obtain. Furthermore, standard unsupervised models like CycleGAN often struggle to maintain speaker identity while improving clarity, frequently introducing artifacts or failing to enhance intelligibility. This research addresses these hurdles by proposing a GAN-based framework that operates effectively on unpaired data.

Methodology

The proposed architecture consists of three primary modules: a Generator ($G$), a Discriminator ($D$), and a Controller ($C$).

High-level architecture of the proposed GAN-based framework, illustrating the interaction between the Generator, Discriminator, and Controller.

The Generator and Discriminator are trained on a large corpus of normal speech (the ASTMIC dataset). The Generator learns to synthesize high-quality, clear speech from a latent "code" ($c$), while the Discriminator learns to distinguish between real and generated normal speech. Because the Generator is trained exclusively on healthy speech, its output is inherently biased toward high intelligibility.

The Controller acts as the bridge, taking impaired speech as input and learning to output the specific code ($c$) required by the Generator to produce a "normal" version of that utterance. A key innovation is the use of "perceptual loss," where activations in the Discriminator's hidden layers are compared to ensure the output matches the input's high-level content and speaker characteristics. Importantly, the Generator remains fixed during Controller training to prevent it from drifting toward producing distorted speech.

Results and Discussion

The model was evaluated using the ASTMIC dataset for normal speech and a dataset of 132 Mandarin utterances from a patient with serious articulatory injuries. Performance was compared against CycleGAN and a supervised Conditional GAN (cGAN) using a healthy proxy speaker.

Mean Opinion Scores (MOS) for content and speaker similarity, showing the proposed model's superior performance compared to cGAN and CycleGAN.

Human listeners rated the proposed model significantly higher in articulation (~59.4%) compared to CycleGAN (~41.4%) and cGAN (~33.2%). As shown in the MOS results, the proposed method also excelled in preserving speaker identity and linguistic content. Spectrogram analysis further revealed that the model successfully reconstructed word boundaries that were blurred in the original impaired speech.

A notable finding was the "steganography" failure in CycleGAN, where the model hides low-level noise to satisfy cycle-consistency requirements, leading to poor audio quality. The proposed one-way mapping avoids this issue. However, limitations remain, such as temporal discontinuities at window boundaries and the use of the Griffin-Lim algorithm for phase reconstruction, which can produce slightly metallic audio.

Conclusion

This work demonstrates a robust approach to restoring impaired speech using unpaired data. By locking the synthesis process to a clear-speech domain and using a controller for feature mapping, the model achieves high intelligibility without sacrificing speaker identity, offering a promising tool for post-operative patient rehabilitation.

Summary updated

BibTeX
@inproceedings{chen2019generative,
  author = {Li-Wei Chen and Hung-yi Lee and Yu Tsao},
  title = {{Generative Adversarial Networks for Unpaired Voice Transformation on Impaired Speech}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1265},
}
Figure