Abstract
Several fully end-to-end text-to-speech (TTS) models have been proposed that have shown better performance compared to cascade models. However, they often generate unstable pitch contour with audible artifacts when the dataset contains emotional attributes, i.e., large diversity of pronunciation and prosody. To address this problem, we propose Period VITS, a novel end-to-end TTS model that incorporates an explicit periodicity generator. In the proposed method, we introduce a frame pitch predictor that predicts prosodic features, such as pitch and voicing flags, from the input text. From these features, the proposed periodicity generator produces a sample-level sinusoidal source that enables the waveform decoder to accurately reproduce the pitch. Finally, the entire model is jointly optimized in an end-to-end manner with variational inference and adversarial objectives. As a result, the decoder becomes capable of generating more stable, expressive, and natural output waveforms. The experimental results showed that the proposed model significantly outperforms baseline models in terms of naturalness, with improved pitch stability in the generated samples.
Introduction
While end-to-end TTS models like VITS have revolutionized speech synthesis by jointly optimizing acoustic modeling and vocoding, they often stumble when faced with "challenging" data. Specifically, when a dataset contains emotional speech—characterized by large swings in pitch and diverse pronunciations—standard models frequently produce unstable pitch contours and audible artifacts. The core issue identifies a specific bottleneck: non-autoregressive vocoders struggle to generate accurate pitch information purely from latent features, leading to "buzzy" or "shaky" voices when high expressivity is required.
Methodology
Period VITS builds upon the original VITS architecture, which utilizes a Variational Autoencoder (VAE) and Normalizing Flows, but introduces three key components to enforce pitch stability:
- Frame Pitch Predictor: Integrated within the prior encoder, this module predicts prosodic features—specifically the fundamental frequency ($F_0$) and voicing flags—directly from the input text at a frame level.
- Periodicity Generator: This module converts the predicted frame-level pitch into a sample-level sinusoidal source signal. This signal serves as a structural "skeleton" for the periodic components of the speech waveform.
- Enhanced Decoder: The waveform decoder is modified to receive this sinusoidal source at every up-sampling stage. By injecting the periodic source directly into the decoder's hidden representations, the model ensures the generated waveform adheres strictly to the intended pitch.
The entire system is trained end-to-end using a combination of VAE reconstruction loss, KL divergence, adversarial (GAN) loss, and a specific pitch reconstruction loss ($L_{pitch}$).
Experimental Setup
The model was evaluated using a large-scale internal dataset of phonetically-balanced emotional Japanese speech. The dataset included 15 professional speakers (5 male, 10 female), totaling approximately 84 hours of audio. Each speaker provided recordings in three styles: neutral, happy, and sad. Audio was sampled at 24 kHz, with $F_0$ extracted using an improved time-frequency trajectory excitation vocoder for ground truth.
Results and Discussion
The researchers conducted Mean Opinion Score (MOS) tests with 18 native Japanese speakers to evaluate naturalness across different emotional styles.
Superior Naturalness and Stability
Period VITS outperformed all baselines, including the original VITS and a high-quality cascade model (FastSpeech 2 + HiFi-GAN). Notably, for neutral and sad styles, Period VITS achieved scores (4.66 and 4.69 respectively) that were statistically indistinguishable from original human recordings.
A critical insight from the study is that simply concatenating pitch features at the frame level is insufficient for high-quality synthesis. Significant improvements only occurred when pitch was converted to a sample-level periodic source. This is particularly evident in "happy" speech, which contains the most prosodic variation and saw the greatest relative gains.
Visual analysis of mel-spectrograms confirms that Period VITS produces much clearer and more stable harmonic structures compared to baseline models like CAT-P-VITS, which often exhibit "swaying" pitch contours.
Conclusion
Period VITS represents a significant advancement in emotional TTS by explicitly modeling periodicity. While highly effective, future work could address reliance on explicit pitch extraction during training by exploring implicit pitch modeling. Additionally, extending the architecture to tonal languages and further refining quality for extreme expressivity remain key areas for development.
Summary updated