TL;DR
This paper investigates how different prosodic contextual factors influence HMM-based speech synthesis quality for English and Japanese. The authors demonstrate that a more compact set of carefully selected contextual factors can achieve comparable synthesis quality to a full set, significantly reducing computation time during model training.
What Problem Does This Paper Address?
Modern HMM-based speech synthesis systems aim to model not just the phonetic sounds but also the prosody (intonation, rhythm, stress) of speech to achieve naturalness. This is often done by incorporating various "contextual factors" into the HMM training process. These factors describe the linguistic and prosodic environment surrounding a given speech unit (like a phoneme). While using a comprehensive set of factors (a "full context") seems intuitive, it leads to a combinatorial explosion of possible contexts, making model training computationally expensive due to the need for parameter tying (e.g., decision-tree-based clustering). This paper addresses the question: can we achieve high-quality speech synthesis with a significantly reduced set of prosodic contextual factors, thereby saving computational resources without sacrificing naturalness?
Methods
The core of this work involves systematically evaluating the impact of different prosodic contextual factors on HMM-based speech synthesis for both English and Japanese.
- Contextual Factor Selection: The authors start with a "full context" set for each language, derived from existing HTS (HMM-based Text-to-Speech) systems. For English, this includes factors related to phonemes, syllables, words, phrases, and utterances. For Japanese, it incorporates mora-level accent information, part-of-speech, breath groups, and sentence length.
- Objective Evaluation: They train HMMs using various subsets of these contextual factors. The quality of the synthesized speech is then objectively measured using:
- Mel-cepstral distance (for spectral distortion)
- Root Mean Square (RMS) error of log F0 (for pitch distortion)
- RMS error of duration (for temporal distortion)
- Subjective Evaluation: To complement objective measures, they conduct listening tests (XAB and MOS) where human listeners rate the naturalness and similarity of synthesized speech to vocoded reference speech.
- Compact Context Creation: Based on objective evaluations, they identify the most impactful factors and progressively build "simple" and "minimum" context sets. The goal is to find the smallest set that yields performance close to the full context.
- Computational Cost Analysis: A key aspect is measuring the reduction in computation time for context clustering when using these more compact context sets.
Datasets
- English Speech Data: CMU ARCTIC database. The authors used speech data from six speakers (four male, two female), noting they are not professional narrators.
- Japanese Speech Data: ATR Japanese speech database set B. This dataset consists of speech from ten professional narrators (six male, four female), each uttering 503 phonetically balanced sentences. The Japanese speech data was manually labeled with accent information.
Key Insights and Ideas
- Not all prosodic factors are created equal: The study reveals that some prosodic contextual factors contribute significantly to synthesis quality (especially F0 reproduction), while others have a minimal impact. For instance, in English, the number of stressed syllables was found to be highly effective.
- Language-specific importance: The relative importance of factors can differ between languages, reflecting their distinct prosodic structures (e.g., mora-based accent in Japanese vs. stress-based accent in English).
- Compact contexts are viable: The research strongly suggests that it's possible to achieve synthesis quality comparable to using all possible contextual factors by selecting a smaller, more judicious set. This is a crucial finding for practical HMM-based TTS system development.
- Computational savings are substantial: Reducing the number of contextual factors leads to significant reductions in the computation time required for context clustering, a bottleneck in HMM training.
Key Results
- English:
- Objective evaluation showed that factors like the number of stressed syllables (G) significantly reduced F0 distortion.
- A "simple context" (13 factors) and a "minimum context" (derived by iteratively adding factors) achieved performance comparable to the "full context."
- Subjective tests indicated that for some speakers, compact contexts were indistinguishable from the full context, while for others, there was a slight preference for the full context, but the difference was often marginal.
- Japanese:
- Accent information was found to be critical for F0 reproducibility.
- Sentence length had a minor positive effect on duration reproducibility.
- A "simple context" (using accent, part of speech, breath group, and sentence length) achieved comparable performance to the "full context" in both objective and subjective evaluations.
- Computation Cost: Using compact contexts (simple/minimum) resulted in significant speedups for context clustering. For example, for spectral features in English, the computation time was reduced from 1877 seconds (full context) to 1332 seconds (simple context), a 29% reduction. For Japanese, spectral computation time dropped from 1007 seconds to 816 seconds (19% reduction).
Limitations and Future Work
- Speaker Variability: The paper notes that speaker variability can be larger for certain features (e.g., duration in Japanese). While they selected speakers with worst objective results for subjective tests, informal listening suggested comparable performance for speakers with better objective scores.
- Manual Annotation: For Japanese, manual labeling of accent information was required, which is labor-intensive.
- Future Work: The authors suggest extending this research to linguistically different types of speech, such as tonal languages, to further validate the findings. They also aim to explore performance evaluation for different speaking styles.
Summary updated