A hierarchical F0 modeling method for HMM-based speech synthesis

Ming Lei, Yi-Jian Wu, Frank K. Soong, Zhen-Hua Ling, Li-Rong DaiInterspeech 2010 · OpenReview

Abstract

The conventional state-based F0 modeling in HMM-based speech synthesis systems is effective at capturing micro-prosodic features but struggles to characterize long-term pitch patterns directly. This paper presents a hierarchical F0 modeling method to address this issue. In this method, different F0 models are used to model the pitch patterns for different prosodic layers (including state, phone, syllable, and word) and are combined with an additive structure. In model training, the F0 model for each layer is first initialized using the residual between original F0s and generated F0s from other layers, and then the F0 models of all layers are re-estimated simultaneously under a minimum generation error (MGE) training framework. Experimental results show that the proposed hierarchical F0 modeling method significantly outperforms the conventional state-based F0 modeling method.

Introduction

In Hidden Markov Model (HMM) based speech synthesis, mapping text to acoustic features like spectrum and fundamental frequency (F0) is a critical task. Traditionally, F0 is modeled at the state level (sub-phonetic units). While this approach captures "micro-prosody"—the rapid fluctuations caused by specific speech sounds—it often fails to represent long-term pitch patterns such as phrase intonation or word stress. This limitation results in synthetic speech that lacks the natural flow and melody of human speech. This paper proposes a hierarchical framework to explicitly model these higher-level linguistic units.

Proposed Methodology

The proposed framework relies on two primary innovations: an additive hierarchical structure and a joint optimization training process.

Additive Structure

The model assumes that a natural F0 contour (on a log scale) is the sum of additive components from different linguistic levels: $$\log F0_{all} = \log F0_{state} + \log F0_{phone} + \log F0_{syllable} + \log F0_{word}$$ Each layer utilizes its own HMM to capture patterns specific to that unit. For instance, the syllable layer focuses on position within a word, while the state layer handles micro-prosodic perturbations.

Training and Optimization

The training process involves two main stages:

  1. Residual Initialization: Models are initialized hierarchically. The highest layer (word) is trained first, followed by the syllable layer using the residual error, and so on down to the state level.
  2. Minimum Generation Error (MGE) Training: To ensure the layers work harmoniously, the authors employ MGE training. Unlike standard Maximum Likelihood (ML) training which treats layers independently, MGE re-estimates all parameters across all layers simultaneously to minimize the difference between the generated and natural pitch contours.
Flowchart of the hierarchical training process, including initialization and joint MGE optimization.

Experimental Results

The system was evaluated using a male English speech database of 3,526 sentences. Objective metrics showed that the Root Mean Square Error (RMSE) of the F0 contour decreased consistently as more layers were added, with the "AddingWord" configuration (all four layers) performing best on both training and test sets.

Subjective listening tests conducted by speech experts further validated the approach. The hierarchical model incorporating all layers was significantly preferred over the state-only baseline.

Subjective preference test results comparing the hierarchical "All layers" model against the baseline.

Key insights from the results include:

  • Structural Memory: Modeling higher-level units provides the system with the "memory" needed to maintain consistent intonation over long spans.
  • Necessity of Joint Optimization: Hierarchical models performed poorly before the MGE step, indicating that joint optimization is crucial for managing the complex interactions between additive layers.

Conclusion

The hierarchical F0 modeling method effectively bridges the gap between micro-prosody and long-term intonation in HMM-based synthesis. Future work will focus on automating the determination of optimal decision tree complexity for each layer and integrating more sophisticated voiced/unvoiced decision-making into the hierarchy.

Summary updated

Abstract

The conventional state-based F0 modeling in HMM-based speech synthesis system is good at capturing micro prosodic features, but difficult to characterize long term pitch patterns directly. This paper presents a hierarchical F0 modeling method to address this issue. In this method, different F0 models are used to model the pitch patterns for different prosodic layers (including state, phone, syllable, word, etc), and are combined with an additive structure. In model training, the F0 model for each layer is firstly initialized by using the residual between original F0s and generated F0s from other layers as training data, and then the F0 models of all layers are re-estimated simultaneously under a minimum generation error (MGE) training framework. We investigate the effectiveness of hierarchical F0 modeling with different layer settings, experimental results show that the proposed hierarchical F0 modeling method significantly outperforms the conventional state-based F0 modeling method.
BibTeX
@inproceedings{lei2010hierarchical,
  author = {Ming Lei and Yi-Jian Wu and Frank K. Soong and Zhen-Hua Ling and Li-Rong Dai},
  title = {{A hierarchical F0 modeling method for HMM-based speech synthesis}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=HSEBoDzS3f},
}
Figure