Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks

Yuki Saito, Shinnosuke Takamichi, Hiroshi Saruwatari

TL;DR

This paper introduces a novel framework for speech synthesis that uses Generative Adversarial Networks (GANs) to combat the "over-smoothing" effect common in neural text-to-speech. By training an acoustic model to "deceive" a discriminator that distinguishes between natural and synthetic speech, the authors produce speech parameters that are statistically much closer to natural human speech, with the Wasserstein GAN (W-GAN) variant emerging as the top performer.

What Problem Does This Paper Address?

In Statistical Parametric Speech Synthesis (SPSS), deep neural networks (DNNs) are trained to map linguistic features (like phonemes) to acoustic parameters (like spectrum and pitch). While effective, these models often suffer from over-smoothing.

Because standard training criteria (like Mean Squared Error) encourage the model to predict the "average" value to minimize error, the resulting speech parameters lose the fine-grained variance and detail found in natural speech. This makes the synthetic voice sound muffled, buzzy, or "robotic." Previous attempts to fix this involved manually adding constraints like Global Variance (GV), but these are often rigid and don't capture the full complexity of human speech.

Methods

The authors propose a GAN-based training framework for acoustic models:

  1. The Generator: This is the standard acoustic model (a DNN) that predicts speech parameters ($y$) from linguistic features ($x$).
  2. The Discriminator: A second DNN trained to distinguish between natural speech parameters and those produced by the generator.
  3. Hybrid Loss Function: Instead of just minimizing the generation error (MGE), the generator is trained on a weighted sum of MGE and an adversarial loss. This adversarial loss forces the generator to produce outputs that the discriminator perceives as "natural."

The researchers experimented with several GAN variants to see which mathematical "distance" between distributions worked best for speech:

  • Standard GAN: Minimizes Jensen–Shannon (JS) divergence.
  • f-GAN: A generalized framework including KL-divergence.
  • Least Squares GAN (LS-GAN): Uses a Pearson $\chi^2$ divergence.
  • Wasserstein GAN (W-GAN): Minimizes the Earth-Mover’s distance, which is known for more stable training.

Datasets

The study utilized the ATR Japanese speech database, specifically:

  • Speaker: One male speaker.
  • Size: 503 phonetically balanced sentences (450 for training, 53 for evaluation).
  • Features: 24th-order mel-cepstral coefficients (spectral parameters), $F_0$ (pitch), and 5 band-aperiodicity excitation parameters.

Key Insights and Ideas

  • Implicit Statistical Modeling: Unlike previous methods that required researchers to explicitly define features like "Global Variance" or "Modulation Spectrum," the GAN learns these natural statistics implicitly. The discriminator acts as a learned, high-dimensional "naturalness" metric.
  • W-GAN is the Winner: The Wasserstein GAN significantly outperformed other variants. The authors suggest that the Earth-Mover’s distance provides a more useful gradient for the generator to follow, especially when the synthetic and natural distributions are very different.
  • The "Anti-Spoofing" Connection: The authors point out that the discriminator is essentially a "voice spoofing detector." By training the synthesizer to beat a spoofing detector, they inherently make the speech more realistic.
  • Feature Functions: They found that using a simple "Identity" function (passing raw parameters to the discriminator) worked better for improving quality than using more complex "Static & Delta" features often used in anti-spoofing research.

Key Results

  • Subjective Quality: In AB preference tests, listeners significantly preferred the GAN-based synthesis over the conventional MGE-trained models for both Text-to-Speech (TTS) and Voice Conversion (VC).
  • Over-smoothing Reduction: Visual analysis (scatter plots and GV curves) confirmed that the GAN-generated parameters were as widely distributed as natural speech, whereas conventional models produced tightly clustered, "safe" averages.
  • MOS Scores: In a Mean Opinion Score (MOS) test, the W-GAN variant achieved the highest score (~3.5), significantly outperforming the standard GAN and other divergences.
  • Versatility: The method proved effective across different architectures, including simple Feed-Forward networks and more complex Long Short-Term Memory (LSTM) networks.

Limitations and Future Work

  • Duration Generation: The GAN approach did not significantly improve phoneme or mora duration. The authors suspect the discriminator has a harder time finding "unnaturalness" in timing compared to spectral features.
  • Hyper-parameter Sensitivity: The weight ($\omega_D$) balancing the generation loss and adversarial loss requires careful tuning.
  • Future Directions: The authors suggest extending this work to direct waveform synthesis (moving beyond parameters) and incorporating linguistic dependencies directly into the discriminator to ensure the "naturalness" is also contextually appropriate.

Summary updated

Abstract

A method for statistical parametric speech synthesis incorporating generative adversarial networks (GANs) is proposed. Although powerful deep neural networks techniques can be applied to artificially synthesize speech waveform, the synthetic speech quality is low compared with that of natural speech. One of the issues causing the quality degradation is an oversmoothing effect often observed in the generated speech parameters. A GAN introduced in this paper consists of two neural networks: a discriminator to distinguish natural and generated samples, and a generator to deceive the discriminator. In the proposed framework incorporating the GANs, the discriminator is trained to distinguish natural and generated speech parameters, while the acoustic models are trained to minimize the weighted sum of the conventional minimum generation loss and an adversarial loss for deceiving the discriminator. Since the objective of the GANs is to minimize the divergence (i.e., distribution difference) between the natural and generated speech parameters, the proposed method effectively alleviates the oversmoothing effect on the generated speech parameters. We evaluated the effectiveness for text-to-speech and voice conversion, and found that the proposed method can generate more natural spectral parameters and F0 than conventional minimum generation error training algorithm regardless of its hyperparameter settings. Furthermore, we investigated the effect of the divergence of various GANs, and found that a Wasserstein GAN minimizing the Earth-Mover's distance works the best in terms of improving the synthetic speech quality.
BibTeX
@article{saito2018statistical,
  author = {Yuki Saito and Shinnosuke Takamichi and Hiroshi Saruwatari},
  title = {{Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=b6xrbMGjUo},
}

Loading related content...

Figure