Novel Inception-GAN for Whispered-to-Normal Speech Conversion

Maitreya Patel, Mihir Parmar, Savan Doshi, Nirmesh Shah, Hemant A. PatilSSW 2019 · DOI

TL;DR

This paper introduces Inception-GAN, a novel Generative Adversarial Network architecture for converting whispered speech to normal speech. By incorporating Inception modules, their model achieves improved stability, reduced computational complexity, and better performance compared to existing CNN-based GANs, making whispered speech conversion more accessible and efficient.

What Problem Does This Paper Address?

Whispered-to-normal speech (WHSP2SPCH) conversion is a challenging task with significant real-world applications, from enhancing privacy in public spaces to aiding individuals with vocal impairments. The core difficulty lies in the fundamental acoustic differences between whispered and normal speech: whispered speech lacks vocal fold vibration (fundamental frequency, or Fo), leading to spectral differences, altered phone durations, and energy distributions. Existing methods, particularly those based on Convolutional Neural Networks (CNNs) within Generative Adversarial Networks (GANs), often struggle with training stability, computational expense, and the risk of overfitting. This makes developing efficient and effective WHSP2SPCH systems a persistent challenge.

Methods

The paper proposes Inception-GAN, a novel architecture that leverages the Inception module within both the generator and discriminator of a GAN. The Inception module, originally designed for computer vision, allows for capturing features at multiple scales simultaneously by using parallel convolutional filters of different sizes (1x1, 3x3, 5x5) and max-pooling. The authors adapt this by using 1x1 convolutions for dimensionality reduction before larger convolutions, and crucially, they replace the expensive 5x5 convolutions with two consecutive 3x3 convolutions. This modification significantly reduces computational complexity while aiming to preserve linguistic information.

The generator in Inception-GAN is composed of stacked Inception modules followed by a single CNN layer. The discriminator also utilizes Inception modules, with average pooling after each to reduce feature map size. For training, they employ a standard adversarial loss function for both the generator and discriminator, aiming to confuse the discriminator with generated speech.

Datasets

The primary dataset used for training and evaluation is the whispered TIMIT (wTIMIT) corpus. The authors selected data from six speakers (three male, three female) for development, using 1164 parallel utterances for training and 35 utterances for testing.

Key Insights and Ideas

  • Inception Modules for Speech: Applying Inception modules, traditionally used in image processing, to speech conversion offers a way to capture multi-scale features efficiently.
  • Computational Efficiency: The modified Inception module, particularly the replacement of 5x5 convolutions with two 3x3 ones, is key to reducing the computational burden of GANs for speech tasks.
  • Training Stability: The Inception-GAN architecture demonstrates improved training stability and is less sensitive to hyperparameters compared to traditional CNN-GANs.
  • Bridging the Gap: The architecture effectively addresses the spectral and Fo differences between whispered and normal speech, leading to more natural-sounding conversions.

Key Results

  • Objective Evaluation: Inception-GAN showed significant improvements over the baseline CNN-GAN. For speaker-specific evaluations, it achieved 8.9% better performance for male speakers and 6.2% for female speakers in terms of Mel Cepstral Distortion (MCD). While RMSE of log(Fo) was comparable, the overall MCD reduction indicates better spectral accuracy.
  • Subjective Evaluation (MOS): In comparative subjective tests, Inception-GAN was preferred by subjects 16.67% more often than CNN-GAN, suggesting a more natural and preferred output quality.
  • Convergence: Inception-GAN converged much faster and more reliably than CNN-GAN, which showed increasing loss functions and never converged within 100 epochs.
  • Global Variance (GV): Plots of Global Variance indicated that Inception-GAN generated MCC features that were closer to the target (original normal speech) compared to CNN-GAN, especially for female speakers.

Limitations and Future Work

  • Limitations: The paper acknowledges that while Inception-GAN improves upon CNN-GAN, there is still room for enhancing the voice quality of converted speech. The current objective functions might not fully capture perceptual differences.
  • Future Work: The authors plan to explore the use of high-quality vocoders like WORLD or Wavenet to further improve the output quality. They also aim to investigate better objective functions that can perceptually optimize GAN parameters for more natural-sounding speech.

Summary updated

BibTeX
@inproceedings{patel2019novel,
  author = {Maitreya Patel and Mihir Parmar and Savan Doshi and Nirmesh Shah and Hemant A. Patil},
  title = {{Novel Inception-GAN for Whispered-to-Normal Speech Conversion}},
  booktitle = {SSW},
  year = {2019},
  doi = {10.21437/ssw.2019-16},
}
Figure