Abstract
In this paper, we propose a variational recurrent neural network (VRNN) based method for modeling and generating speech parameter sequences. In recent years, the performance of speech synthesis systems has been improved over conventional techniques thanks to deep learning-based acoustic models. Among the popular deep learning techniques, recurrent neural networks (RNNs) have been successful in modeling time-dependent sequential data efficiently. However, due to the deterministic nature of RNNs' prediction, such models do not reflect the full complexity of highly structured data, like natural speech. In this regard, we propose the adversarially trained variational recurrent neural network (AdVRNN) which uses VRNN to better represent the variability of natural speech for acoustic modeling in speech synthesis. Also, we apply an adversarial learning scheme in training AdVRNN to overcome the oversmoothing problem. We conducted comparative experiments for the proposed VRNN with the conventional gated recurrent unit (GRU) for a speech synthesis system. It is shown that the proposed AdVRNN-based method performed better than the conventional GRU technique.
Introduction
Traditional Statistical Parametric Speech Synthesis (SPSS) often relies on deterministic Recurrent Neural Networks (RNNs) like LSTMs or GRUs. While these architectures excel at modeling temporal sequences, their deterministic nature is a limitation: given the same input, they produce the exact same output. This fails to capture the inherent variability and "texture" of natural human speech, often resulting in robotic or muffled audio.
Variational models, such as the Variational Recurrent Neural Network (VRNN), were introduced to address this by incorporating latent variables that represent natural variability. However, these models frequently suffer from "oversmoothing." Because they aim to maximize the Evidence Lower Bound (ELBO), they often produce an "average" of possible sounds, which humans perceive as blurry or muffled. This research introduces the AdVRNN to inject realism back into variational acoustic models.
Methodology
The proposed Adversarially Trained Variational Recurrent Neural Network (AdVRNN) integrates the temporal power of RNNs with the stochastic flexibility of Variational Autoencoders (VAEs), further refined by a Generative Adversarial Network (GAN) framework.
The VRNN Core and Modified State Updates
Unlike a standard VAE, which treats data points as independent, the VRNN uses an RNN state to track dependencies over time. To prevent error propagation during the synthesis stage, the authors modified the state update equation to remove dependency on the previous generated acoustic feature, relying instead on linguistic features and latent variables.
Linguistic Conditioning and Generation
The model is specifically tailored for speech synthesis by conditioning the decoder on linguistic features (text/phonemes). The generation process produces acoustic features $x_t$—including mel-cepstrum, pitch, and aperiodicity—by integrating the current hidden state, sampled latent variables, and linguistic context.
Adversarial Training
The "secret sauce" of this approach is the adversarial training layer. By introducing a Discriminator network, the training process becomes a competition: the generator tries to produce acoustic sequences indistinguishable from real human speech, while the discriminator tries to identify the synthetic samples. This adversarial loss acts as a corrective force against the blurring effect of the standard variational loss, pushing the model to generate sharp, high-frequency details.
Experimental Results
The researchers evaluated the AdVRNN using a Korean speech database featuring a professional male speaker (2,250 utterances). The AdVRNN was compared against a deep hybrid GRU-based deterministic system.
- Objective Metrics: AdVRNN achieved a lower Mel-Cepstral Distance (MCD) of 5.808 dB compared to 6.203 dB for the GRU, indicating a spectral shape much closer to real speech.
- Subjective Quality: In Mean Opinion Score (MOS) listening tests, the AdVRNN significantly outperformed the baseline, scoring 3.623 against the GRU's 2.836. Listeners found the AdVRNN output to be more intelligible and natural.
Conclusion and Future Work
The AdVRNN successfully leverages stochasticity to represent the variability of natural speech while using adversarial training to solve the oversmoothing problem inherent in variational models. While the model performed slightly worse in pitch ($f_0$) RMSE due to normalization challenges in unvoiced regions, it represents a significant step forward in natural-sounding speech synthesis. Future work will focus on improving pitch contour modeling and exploring more complex discriminator architectures to further refine audio sharpness.
Summary updated