Unsupervised Prosodic Labeling of Speech Synthesis Databases Using Context-Dependent HMMs

Chen-Yu Yang, Zhen-Hua Ling, Li-Rong DaiIEICE Trans. Information and Systems 2014 · OpenReview

Abstract

This paper proposes an automatic and unsupervised method using context-dependent hidden Markov models (CD-HMMs) for the prosodic labeling of speech synthesis databases. The method involves three main steps: initialization, model training, and prosodic labeling. Initial labels are obtained through unsupervised clustering of acoustic features. Subsequently, CD-HMMs for spectral parameters, F0, and phone durations are estimated. These labels are iteratively refined using Viterbi decoding under the maximum likelihood criterion until convergence. Evaluated on Mandarin speech synthesis databases, the method demonstrates high accuracy in labeling prosodic phrase boundaries and emphasis expressions, performing comparably to manual labeling in unit selection speech synthesis systems.

Introduction

To build high-quality, natural-sounding text-to-speech (TTS) systems, computers require prosodic labeling—information about where a speaker pauses and which words they emphasize. Traditionally, this is a manual process that is slow, expensive, and prone to inconsistency. While supervised learning can automate this, it requires large amounts of pre-labeled data for every new speaker or language. This paper addresses this bottleneck with an unsupervised framework that learns prosodic patterns directly from raw audio and text transcripts.

Methodology

The proposed method utilizes an iterative "initialization and refinement" framework to automatically generate prosodic labels.

Flowchart of the proposed unsupervised prosodic labeling framework, illustrating the iterative cycle between model training and Viterbi-based labeling.

1. Initialization

The process starts with an acoustic-based "best guess." For prosodic phrase boundaries, the system analyzes the duration of silences between words. Using unsupervised clustering, it separates short word-level pauses from longer phrase-level pauses.

Detailed flowchart for the initialization of prosodic phrase boundary labels using acoustic parameter extraction and unsupervised clustering.

For emphasis expression, the system compares the pitch (F0) of the natural recording against a neutral synthetic version; words with significantly higher pitch are initially flagged as emphasized.

2. Model Training

Using the initial labels, the system trains Context-Dependent Hidden Markov Models (CD-HMMs). These models capture how spectral parameters, pitch, and duration change based on the surrounding phonetic and prosodic context.

3. Prosodic Labeling and Iteration

The system then performs Viterbi decoding to find the sequence of labels that best fits the trained models and the actual audio. This process is repeated: as labels become more accurate, the HMMs improve, which in turn makes the labeling even more precise. In Mandarin phrase boundary tests, the F-score improved from 62.7% after initialization to 72.5% upon convergence.

Key Insights and Results

  • Acoustics Over Text: The study found that acoustic-based labeling is far more accurate than text-analysis-based methods (which predict pauses based on grammar). Human speech is idiosyncratic; speakers often pause for breath or effect in ways that grammatical rules cannot predict.
  • The Power of Iteration: The iterative refinement significantly boosts performance, proving that CD-HMMs can effectively learn a speaker's unique prosodic style from unlabeled data.
  • Phrase Boundary Accuracy: The unsupervised method achieved a 72.5% F-score, remarkably close to supervised methods (76.1%) and far superior to text-based methods (49.2%).
  • Subjective Naturalness: In Mean Opinion Score (MOS) tests, listeners rated speech synthesized using unsupervised labels as nearly identical in quality to speech synthesized using manual, human-provided labels.
  • Emphasis Perception: In perception tests, 96% of the words the system intended to emphasize were correctly perceived as emphasized by human listeners.

Conclusion

This unsupervised approach effectively removes the manual labeling bottleneck in TTS database construction. While successful, future work could involve incorporating more complex acoustic cues like "pitch reset," using non-uniform priors for label distributions, and exploring deep learning architectures to further enhance acoustic modeling of complex prosodic patterns.

Summary updated

Abstract

In this paper, an automatic and unsupervised method using context-dependent hidden Markov models (CD-HMMs) is proposed for the prosodic labeling of speech synthesis databases. This method consists of three main steps, i.e., initialization, model training and prosodic labeling. The initial prosodic labels are obtained by unsupervised clustering using the acoustic features designed according to the characteristics of the prosodic descriptor to be labeled. Then, CD-HMMs of the spectral parameters, F0s and phone durations are estimated by a means similar to the HMM-based parametric speech synthesis using the initial prosodic labels. These labels are further updated by Viterbi decoding under the maximum likelihood criterion given the acoustic feature sequences and the trained CD-HMMs. The model training and prosodic labeling procedures are conducted iteratively until convergence. The performance of the proposed method is evaluated on Mandarin speech synthesis databases and two prosodic descriptors are investigated, i.e., the prosodic phrase boundary and the emphasis expression. In our implementation, the prosodic phrase boundary labels are initialized by clustering the durations of the pauses between every two consecutive prosodic words, and the emphasis expression labels are initialized by examining the differences between the original and the synthetic F0 trajectories. Experimental results show that the proposed method is able to label the prosodic phrase boundary positions much more accurately than the text-analysis-based method without requiring any manually labeled training data. The unit selection speech synthesis system constructed using the prosodic phrase boundary labels generated by our proposed method achieves similar performance to that using the manual labels. Furthermore, the unit selection speech synthesis system constructed using the emphasis expression labels generated by our proposed method can convey the emphasis information effectively while maintaining the naturalness of synthetic speech.
BibTeX
@article{yang2014unsupervised,
  author = {Chen-Yu Yang and Zhen-Hua Ling and Li-Rong Dai},
  title = {{Unsupervised Prosodic Labeling of Speech Synthesis Databases Using Context-Dependent HMMs}},
  journal = {IEICE Trans. Information and Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=yhB0AjKP69},
}
Figure