Acoustic Modeling Using Adversarially Trained Variational Recurrent Neural Network for Speech Synthesis

Joun Yeop Lee, Sung Jun Cheon, Byoung Jin Choi, Nam Soo Kim, Eunwoo SongInterspeech 2018 · OpenReview

Abstract

In this paper, we propose a variational recurrent neural network (VRNN) based method for modeling and generating speech parameter sequences. In recent years, the performance of speech synthesis systems has been improved over conventional techniques thanks to deep learning-based acoustic models. Among the popular deep learning techniques, recurrent neural networks (RNNs) have been successful in modeling time-dependent sequential data efficiently. However, due to the deterministic nature of RNNs' prediction, such models do not reflect the full complexity of highly structured data, like natural speech. In this regard, we propose the adversarially trained variational recurrent neural network (AdVRNN) which uses VRNN to better represent the variability of natural speech for acoustic modeling in speech synthesis. Also, we apply an adversarial learning scheme in training AdVRNN to overcome the oversmoothing problem. We conducted comparative experiments for the proposed VRNN with the conventional gated recurrent unit (GRU) for a speech synthesis system. It is shown that the proposed AdVRNN-based method performed better than the conventional GRU technique.

Introduction

Traditional Statistical Parametric Speech Synthesis (SPSS) often relies on deterministic Recurrent Neural Networks (RNNs) like LSTMs or GRUs. While these architectures excel at modeling temporal sequences, their deterministic nature is a limitation: given the same input, they produce the exact same output. This fails to capture the inherent variability and "texture" of natural human speech, often resulting in robotic or muffled audio.

Variational models, such as the Variational Recurrent Neural Network (VRNN), were introduced to address this by incorporating latent variables that represent natural variability. However, these models frequently suffer from "oversmoothing." Because they aim to maximize the Evidence Lower Bound (ELBO), they often produce an "average" of possible sounds, which humans perceive as blurry or muffled. This research introduces the AdVRNN to inject realism back into variational acoustic models.

Methodology

The proposed Adversarially Trained Variational Recurrent Neural Network (AdVRNN) integrates the temporal power of RNNs with the stochastic flexibility of Variational Autoencoders (VAEs), further refined by a Generative Adversarial Network (GAN) framework.

The VRNN Core and Modified State Updates

Unlike a standard VAE, which treats data points as independent, the VRNN uses an RNN state to track dependencies over time. To prevent error propagation during the synthesis stage, the authors modified the state update equation to remove dependency on the previous generated acoustic feature, relying instead on linguistic features and latent variables.

The recurrence process in the AdVRNN updates the hidden state $h_t$ using the previous state $h_{t-1}$, the latent variable $z_t$, and linguistic features $l_t$.

Linguistic Conditioning and Generation

The model is specifically tailored for speech synthesis by conditioning the decoder on linguistic features (text/phonemes). The generation process produces acoustic features $x_t$—including mel-cepstrum, pitch, and aperiodicity—by integrating the current hidden state, sampled latent variables, and linguistic context.

The generation process produces acoustic features $x_t$ conditioned on the hidden state $h_{t-1}$, latent variable $z_t$, and linguistic features $l_t$.

Adversarial Training

The "secret sauce" of this approach is the adversarial training layer. By introducing a Discriminator network, the training process becomes a competition: the generator tries to produce acoustic sequences indistinguishable from real human speech, while the discriminator tries to identify the synthetic samples. This adversarial loss acts as a corrective force against the blurring effect of the standard variational loss, pushing the model to generate sharp, high-frequency details.

Experimental Results

The researchers evaluated the AdVRNN using a Korean speech database featuring a professional male speaker (2,250 utterances). The AdVRNN was compared against a deep hybrid GRU-based deterministic system.

  • Objective Metrics: AdVRNN achieved a lower Mel-Cepstral Distance (MCD) of 5.808 dB compared to 6.203 dB for the GRU, indicating a spectral shape much closer to real speech.
  • Subjective Quality: In Mean Opinion Score (MOS) listening tests, the AdVRNN significantly outperformed the baseline, scoring 3.623 against the GRU's 2.836. Listeners found the AdVRNN output to be more intelligible and natural.

Conclusion and Future Work

The AdVRNN successfully leverages stochasticity to represent the variability of natural speech while using adversarial training to solve the oversmoothing problem inherent in variational models. While the model performed slightly worse in pitch ($f_0$) RMSE due to normalization challenges in unvoiced regions, it represents a significant step forward in natural-sounding speech synthesis. Future work will focus on improving pitch contour modeling and exploring more complex discriminator architectures to further refine audio sharpness.

Summary updated

Abstract

In this paper, we propose a variational recurrent neural network (VRNN) based method for modeling and generating speech parameter sequences. In recent years, the performance of speech synthesis systems has been improved over conventional techniques thanks to deep learning-based acoustic models. Among the popular deep learning techniques, recurrent neural networks (RNNs) has been successful in modeling time-dependent sequential data efficiently. However, due to the deterministic nature of RNNs prediction, such models do not reflect the full complexity of highly structured data, like natural speech. In this regard, we propose adversarially trained variational recurrent neural network (AdVRNN) which use VRNN to better represent the variability of natural speech for acoustic modeling in speech synthesis. Also, we apply adversarial learning scheme in training AdVRNN to overcome oversmoothing problem. We conducted comparative experiments for the proposed VRNN with the conventional gated recurrent unit which is one of RNNs, for speech synthesis system. It is shown that the proposed AdVRNN based method performed better than the conventional GRU technique.
BibTeX
@inproceedings{lee2018acoustic,
  author = {Joun Yeop Lee and Sung Jun Cheon and Byoung Jin Choi and Nam Soo Kim and Eunwoo Song},
  title = {{Acoustic Modeling Using Adversarially Trained Variational Recurrent Neural Network for Speech Synthesis}},
  booktitle = {Interspeech},
  year = {2018},
  url = {https://openreview.net/forum?id=6CHoPspEzA},
}
Figure