Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks

Yuki Saito, Shinnosuke Takamichi, Hiroshi Saruwatari

Abstract

A method for statistical parametric speech synthesis incorporating generative adversarial networks (GANs) is proposed. Although powerful deep neural networks (DNNs) techniques can be applied to artificially synthesize speech waveform, the synthetic speech quality is low compared with that of natural speech. One of the issues causing the quality degradation is an over-smoothing effect often observed in the generated speech parameters. A GAN introduced in this paper consists of two neural networks: a discriminator to distinguish natural and generated samples, and a generator to deceive the discriminator. In the proposed framework incorporating the GANs, the discriminator is trained to distinguish natural and generated speech parameters, while the acoustic models are trained to minimize the weighted sum of the conventional minimum generation loss and an adversarial loss for deceiving the discriminator. Since the objective of the GANs is to minimize the divergence (i.e., distribution difference) between the natural and generated speech parameters, the proposed method effectively alleviates the over-smoothing effect on the generated speech parameters. We evaluated the effectiveness for text-to-speech and voice conversion, and found that the proposed method can generate more natural spectral parameters and $F_0$ than conventional minimum generation error training algorithm regardless its hyper-parameter settings. Furthermore, we investigated the effect of the divergence of various GANs, and found that a Wasserstein GAN minimizing the Earth-Mover's distance works the best in terms of improving synthetic speech quality.

Introduction

In Statistical Parametric Speech Synthesis (SPSS), deep neural networks (DNNs) are typically trained to map linguistic features, such as phonemes, to acoustic parameters like spectrum and pitch. While this approach has advanced the field, it frequently suffers from a phenomenon known as over-smoothing.

Standard training criteria, such as Mean Squared Error (MSE), encourage models to predict the conditional mean of the distribution to minimize error. Consequently, the generated speech parameters often lack the fine-grained variance and detail present in natural speech, leading to synthetic voices that sound muffled or "robotic." While previous methods like Global Variance (GV) constraints attempted to address this, they are often too rigid to capture the full complexity of human speech. This paper proposes a Generative Adversarial Network (GAN) framework to implicitly learn these natural statistics and alleviate over-smoothing.

Methodology

The proposed framework utilizes a GAN-based training approach for acoustic models, consisting of two primary components:

  1. The Generator: A standard acoustic model (DNN) that predicts speech parameters ($\hat{y}$) from linguistic features ($x$).
  2. The Discriminator: A secondary DNN trained to distinguish between natural speech parameters ($y$) and the generated parameters ($\hat{y}$).

The training process is split into two phases. First, the discriminator is trained to maximize its ability to correctly classify natural and synthetic samples.

Training process of the discriminator to distinguish natural and generated speech parameters.

Second, the generator is trained using a hybrid loss function. This function combines the conventional minimum generation error (MGE) with an adversarial loss. The adversarial loss encourages the generator to produce outputs that the discriminator classifies as "natural," thereby forcing the synthetic distribution to match the natural one.

Training process of the generator (acoustic models) to minimize both generation and adversarial losses.

The researchers explored several GAN variants to determine the most effective divergence measure for speech synthesis:

  • Standard GAN: Minimizes Jensen–Shannon (JS) divergence.
  • f-GAN: A generalized framework including KL-divergence.
  • Least Squares GAN (LS-GAN): Employs Pearson $\chi^2$ divergence.
  • Wasserstein GAN (W-GAN): Minimizes the Earth-Mover’s distance, offering more stable training gradients.

Experimental Setup

The study used the ATR Japanese speech database, featuring:

  • Speaker: One male speaker.
  • Data: 503 phonetically balanced sentences (450 for training, 53 for evaluation).
  • Acoustic Features: 24th-order mel-cepstral coefficients, $F_0$ (pitch), and 5 band-aperiodicity excitation parameters.

Key Insights

  • Implicit Statistical Modeling: The GAN learns natural speech statistics implicitly through the discriminator, which acts as a learned, high-dimensional "naturalness" metric. This avoids the need for manually defined features like Global Variance.
  • Superiority of W-GAN: The Wasserstein GAN variant significantly outperformed others. The Earth-Mover’s distance provides more informative gradients, particularly when the synthetic and natural distributions have little overlap.
  • Anti-Spoofing Parallels: The discriminator functions similarly to a voice spoofing detector. By training the synthesizer to "deceive" this detector, the realism of the generated speech is inherently improved.
  • Feature Selection: Simple "Identity" functions (passing raw parameters) were found to be more effective for quality improvement than complex "Static & Delta" features typically used in anti-spoofing research.

Results

Subjective evaluations, including AB preference tests and Mean Opinion Score (MOS) tests, demonstrated that listeners significantly preferred GAN-based synthesis over conventional MGE-trained models for both Text-to-Speech (TTS) and Voice Conversion (VC).

Visual analysis of generated parameters confirmed a substantial reduction in over-smoothing; GAN-generated parameters exhibited a distribution width comparable to natural speech, whereas conventional models produced tightly clustered averages. The W-GAN variant achieved the highest MOS (~3.5), outperforming the standard GAN. Furthermore, the method proved robust across different neural architectures, including Feed-Forward and LSTM networks.

Conclusion and Future Work

While the GAN framework effectively improves spectral and pitch parameters, it did not significantly enhance phoneme or mora duration generation. This suggests that the discriminator may find it more challenging to identify "unnaturalness" in temporal patterns compared to spectral features. Future research will focus on extending this framework to direct waveform synthesis and incorporating linguistic dependencies into the discriminator to ensure contextual naturalness.

Summary updated

Abstract

A method for statistical parametric speech synthesis incorporating generative adversarial networks (GANs) is proposed. Although powerful deep neural networks (DNNs) techniques can be applied to artificially synthesize speech waveform, the synthetic speech quality is low compared with that of natural speech. One of the issues causing the quality degradation is an over-smoothing effect often observed in the generated speech parameters. A GAN introduced in this paper consists of two neural networks: a discriminator to distinguish natural and generated samples, and a generator to deceive the discriminator. In the proposed framework incorporating the GANs, the discriminator is trained to distinguish natural and generated speech parameters, while the acoustic models are trained to minimize the weighted sum of the conventional minimum generation loss and an adversarial loss for deceiving the discriminator. Since the objective of the GANs is to minimize the divergence (i.e., distribution difference) between the natural and generated speech parameters, the proposed method effectively alleviates the over-smoothing effect on the generated speech parameters. We evaluated the effectiveness for text-to-speech and voice conversion, and found that the proposed method can generate more natural spectral parameters and $F_0$ than conventional minimum generation error training algorithm regardless its hyper-parameter settings. Furthermore, we investigated the effect of the divergence of various GANs, and found that a Wasserstein GAN minimizing the Earth-Mover's distance works the best in terms of improving synthetic speech quality.
BibTeX
@article{saito2018statistical,
  author = {Yuki Saito and Shinnosuke Takamichi and Hiroshi Saruwatari},
  title = {{Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=b6xrbMGjUo},
}
Figure