Abstract
This paper proposes an automatic and unsupervised method using context-dependent hidden Markov models (CD-HMMs) for the prosodic labeling of speech synthesis databases. The method involves three main steps: initialization, model training, and prosodic labeling. Initial labels are obtained through unsupervised clustering of acoustic features. Subsequently, CD-HMMs for spectral parameters, F0, and phone durations are estimated. These labels are iteratively refined using Viterbi decoding under the maximum likelihood criterion until convergence. Evaluated on Mandarin speech synthesis databases, the method demonstrates high accuracy in labeling prosodic phrase boundaries and emphasis expressions, performing comparably to manual labeling in unit selection speech synthesis systems.
Introduction
To build high-quality, natural-sounding text-to-speech (TTS) systems, computers require prosodic labeling—information about where a speaker pauses and which words they emphasize. Traditionally, this is a manual process that is slow, expensive, and prone to inconsistency. While supervised learning can automate this, it requires large amounts of pre-labeled data for every new speaker or language. This paper addresses this bottleneck with an unsupervised framework that learns prosodic patterns directly from raw audio and text transcripts.
Methodology
The proposed method utilizes an iterative "initialization and refinement" framework to automatically generate prosodic labels.
1. Initialization
The process starts with an acoustic-based "best guess." For prosodic phrase boundaries, the system analyzes the duration of silences between words. Using unsupervised clustering, it separates short word-level pauses from longer phrase-level pauses.
For emphasis expression, the system compares the pitch (F0) of the natural recording against a neutral synthetic version; words with significantly higher pitch are initially flagged as emphasized.
2. Model Training
Using the initial labels, the system trains Context-Dependent Hidden Markov Models (CD-HMMs). These models capture how spectral parameters, pitch, and duration change based on the surrounding phonetic and prosodic context.
3. Prosodic Labeling and Iteration
The system then performs Viterbi decoding to find the sequence of labels that best fits the trained models and the actual audio. This process is repeated: as labels become more accurate, the HMMs improve, which in turn makes the labeling even more precise. In Mandarin phrase boundary tests, the F-score improved from 62.7% after initialization to 72.5% upon convergence.
Key Insights and Results
- Acoustics Over Text: The study found that acoustic-based labeling is far more accurate than text-analysis-based methods (which predict pauses based on grammar). Human speech is idiosyncratic; speakers often pause for breath or effect in ways that grammatical rules cannot predict.
- The Power of Iteration: The iterative refinement significantly boosts performance, proving that CD-HMMs can effectively learn a speaker's unique prosodic style from unlabeled data.
- Phrase Boundary Accuracy: The unsupervised method achieved a 72.5% F-score, remarkably close to supervised methods (76.1%) and far superior to text-based methods (49.2%).
- Subjective Naturalness: In Mean Opinion Score (MOS) tests, listeners rated speech synthesized using unsupervised labels as nearly identical in quality to speech synthesized using manual, human-provided labels.
- Emphasis Perception: In perception tests, 96% of the words the system intended to emphasize were correctly perceived as emphasized by human listeners.
Conclusion
This unsupervised approach effectively removes the manual labeling bottleneck in TTS database construction. While successful, future work could involve incorporating more complex acoustic cues like "pitch reset," using non-uniform priors for label distributions, and exploring deep learning architectures to further enhance acoustic modeling of complex prosodic patterns.
Summary updated