TL;DR
This paper introduces a novel method for synthesizing expressive speech by representing expressiveness as a continuous point in a multi-dimensional space, outperforming traditional discrete label approaches. This allows for richer and more nuanced expressive speech synthesis, particularly for complex data like audiobooks.
What Problem Does This Paper Address?
Generating expressive speech, especially for rich sources like audiobooks, is a significant challenge in text-to-speech (TTS) systems. While audiobooks offer a wealth of expressive nuances, manually annotating this expressiveness is costly and prone to low inter-annotator agreement. Previous unsupervised methods have relied on discrete expressive labels, which may not capture the full diversity of human expression. This paper aims to bridge this gap by exploring more sophisticated representations of expressiveness and how to effectively incorporate them into statistical parametric speech synthesis systems.
Methods
The core contribution is the application and extension of Cluster Adaptive Training (CAT) for expressive speech synthesis. Unlike previous methods that used discrete expressive labels, this work proposes representing expressiveness as a continuous point within a multi-dimensional "expressive-speech space."
- CAT Model: The CAT model, originally developed for speech recognition, is adapted for TTS. It uses a set of cluster models, each with Gaussian mean parameters, to represent different expressive states.
- Continuous Expressive Space: Instead of assigning an utterance to a discrete cluster, each utterance is mapped to a specific point (a weight vector) in this continuous space. This allows for finer-grained representation of expressiveness.
- Utterance-Based CAT: The paper introduces an "utterance-based CAT" approach where each utterance is treated as an individual point in the expressive space.
- Prior Smoothing: To address potential data sparseness issues with short utterances in the continuous space, a prior smoothing technique (borrowed from speaker adaptation) is applied, using discrete label information to regularize the estimates.
- Decision Trees: The CAT system allows for cluster-specific decision trees, enabling more complex modeling of context-dependent expressiveness compared to systems that use a single decision tree.
Datasets
- Training/Evaluation Data: The primary dataset used is audiobook data, specifically "A Tramp Abroad," read by John Greenman.
- 56 chapters were used: 51 for training and 5 for evaluation.
- The data was segmented into narration, carrier, and direct speech, yielding approximately 4.8k utterances.
- A 100% word accuracy was achieved against the book text for model training.
- The average training utterance length was 6.8 seconds.
- Acoustic features included 40 mel-cepstral coefficients, logF0, BAP, and their delta/delta-delta information.
Key Insights and Ideas
- Continuous Representation is Key: The most significant finding is that a continuous representation of expressiveness, mapped to a point in a multi-dimensional space, significantly outperforms discrete labels for audiobook data. This suggests that audiobook expressiveness is too rich and varied to be adequately captured by a limited set of discrete categories.
- CAT's Adaptability: CAT's ability to define cluster-specific decision trees makes it a powerful tool for modeling complex, context-dependent expressive variations.
- Bridging Discrete and Continuous: The use of prior smoothing, leveraging discrete label information to inform the continuous representation, offers a practical way to balance expressiveness and robustness, especially for shorter utterances.
- Objective Measures Align with Subjective: Objective measures like Linear Discriminant Analysis (LDA) on the CAT weight vectors correlate well with subjective listening tests, providing a way to evaluate the quality of expressive representations.
Key Results
- Expressiveness Improvement: The CAT system, using both discrete and continuous representations, significantly outperformed traditional decision tree (DT) and Average Expression Speech Synthesis (AESS) methods in expressiveness tests (Table 1).
- Voice Quality Improvement: The CAT system also showed significant improvements in voice quality compared to DT and AESS (Table 2).
- Continuous vs. Discrete: The continuous, utterance-based CAT system yielded further significant gains in expressiveness over the discrete label-based CAT system (Table 4), highlighting the limitations of discrete labels for complex expressive data.
- Feature Set Impact: For clustering, feature set 'C' (likely related to logF0) showed the best performance in both subjective listening tests and objective LDA measures, indicating its importance for capturing expressive characteristics.
Limitations and Future Work
- Automatic Label Derivation: The current experiments assume that the expressive labels or the point in the expressive space are known. A major direction for future work is to develop methods for automatically deriving this information from text.
- Voice Quality Assessment: While preliminary experiments on voice quality for the continuous system were promising, results were varied. The authors note that listeners tended to prefer more neutral speech when context was lacking, and suggest that appropriate ways to assess voice quality in expressive synthesis need further investigation.
- Parameter Tuning: The smoothing parameter $\tau$ was set to a relatively small value (5) without specific tuning for the task, suggesting potential for further optimization.
Summary updated