TL;DR
This paper introduces a novel semi-supervised approach to modeling prosody in speech synthesis, particularly for languages like Japanese where pitch accent is crucial but costly to annotate. By leveraging Deep Gaussian Processes (DGPs) and a latent variable model, they can learn natural-sounding prosody even with partially annotated training data, achieving performance comparable to fully supervised methods.
What Problem Does This Paper Address?
Generating natural-sounding synthetic speech hinges on accurately modeling prosody – the rhythm, stress, and intonation of speech. In statistical Text-to-Speech (TTS) systems, this often requires meticulously annotated prosodic labels (like pitch accent, duration, and energy) for training data. Manually creating these labels is time-consuming, expensive, and prone to annotator inconsistencies. While end-to-end TTS models have shown promise in automatically learning prosody, they can struggle with languages like Japanese, where pitch accent significantly impacts meaning and isn't explicitly encoded in text. This paper tackles the challenge of building high-quality prosody models with limited annotated data, aiming to reduce the annotation burden while maintaining naturalness.
Methods
The core of this work is the application of Deep Gaussian Processes (DGPs) within a semi-supervised learning framework. DGPs are powerful Bayesian generative models that can capture complex, hierarchical relationships. Here, they are used to model prosodic information as a latent variable.
The proposed method works as follows:
- Latent Variable Representation: Prosodic information (specifically, mora-level and accent-phrase-level pitch accent for Japanese) is represented by latent variables. These latent variables act as a compressed, learned representation of the prosody.
- Deep Gaussian Processes (DGPs): DGPs are employed to model the relationship between linguistic features (text content) and acoustic features (like pitch, spectral information) through these latent variables. The "deep" aspect allows for a more expressive modeling of complex prosodic patterns compared to single-layer models.
- Semi-Supervised Learning with DSVI: The authors adapt a doubly stochastic variational inference (DSVI) technique for DGPs. This allows them to train the model even when some training data is "partially unobserved" – meaning the prosodic labels are missing. The model learns by inferring posterior distributions of these latent variables from both observed acoustic and linguistic features, and also from the partially observed prosodic information.
- Training Strategy: A three-phase training strategy is used: first, train DGP parameters on fully observed data; second, train latent variable parameters for unobserved data; and finally, train both simultaneously.
Datasets
The experiments were conducted using the Japanese speech data of a female speaker (F009) from the XIMERA corpus.
- Total Data: 1533 utterances (approx. 119 minutes).
- Training Data Split:
- Fully Observed: 99 utterances (used as a validation set).
- Partially Unobserved: 1434 utterances (90% of training data, with prosodic labels omitted).
- Test Data: 60 utterances.
The paper also compares their proposed method against baselines trained on:
- FULL: All 1533 utterances with fully available prosodic labels (ideal scenario).
- LABELED: Only the 99 fully labeled utterances.
- W/O ACCENT: All 1533 utterances but without using accent information as context.
Key Insights and Ideas
- Latent variables as prosody proxies: The paper demonstrates that latent variables can effectively capture complex prosodic information, acting as a powerful substitute for explicit, manually annotated labels.
- DGP's expressiveness for prosody: The use of Deep Gaussian Processes, rather than single-layer GPLVMs, is crucial for modeling the intricate patterns found in prosody, especially pitch contours.
- Semi-supervised learning bridges the gap: The DSVI-based semi-supervised approach successfully leverages partially labeled data, significantly reducing the reliance on fully annotated datasets without sacrificing quality.
- Pitch accent is critical for Japanese: The results highlight that even with a large amount of data, omitting pitch accent information (as in the "W/O ACCENT" baseline) leads to degraded prosody, underscoring its importance for naturalness in Japanese TTS.
Key Results
- Objective Evaluation (Table 1): The proposed method (PROPOSED) achieved comparable mel-cepstral distortion (MCEP) and phone duration RMSE to the ideal "FULL" condition. Crucially, it significantly reduced the RMSE of log F0 compared to the "W/O ACCENT" baseline, showing a 19 cent improvement. The "LABELED" method (using only 99 utterances) performed worse objectively.
- Subjective Evaluation (Figure 3): The Mean Opinion Score (MOS) for the PROPOSED method was comparable to the "FULL" condition and significantly higher than both "LABELED" and "W/O ACCENT" methods. This indicates that the synthesized speech is perceived as more natural.
- F0 Contour Analysis (Figure 4): Generated F0 contours from the PROPOSED method closely resembled those from the "FULL" condition, while the "W/O ACCENT" method produced flatter contours, confirming the importance of accent information.
Limitations and Future Work
- Limitations: The paper acknowledges that while the proposed method performs well, it is still an approximation. The training process for latent variables can be sensitive to initialization. The current study focuses on a single speaker and a specific dialect (Tokyo dialect).
- Future Work: The authors suggest extending the approach to more diverse speech data, including tone languages and stress accent languages. They also aim to investigate the effectiveness of their method in low-resource languages and dialects where prosody annotation is particularly challenging.
Summary updated