Improving Mandarin segmental duration prediction with automatically extracted syntax features

Miaomiao Wen, Miaomiao Wang, Keikichi Hirose, Nobuaki MinematsuInterspeech 2010

TL;DR

This paper introduces a novel approach to improve Mandarin speech synthesis by incorporating automatically extracted syntax features into duration prediction models. By leveraging a Chinese syntax parser, the authors demonstrate that these syntax features can significantly enhance the accuracy of predicting phone and pause durations, leading to more natural-sounding synthesized speech.

What Problem Does This Paper Address?

Accurate prediction of segmental duration (how long each sound or pause lasts) is crucial for natural and intelligible speech synthesis. While previous research has acknowledged the link between syntax and prosody, the direct use of syntax features for duration prediction, especially in Mandarin, has been limited. Manually extracting syntax information is time-consuming and requires specialized expertise. This paper addresses the challenge of effectively integrating syntax information into duration prediction models for Mandarin, aiming to improve synthesis quality without the burden of manual feature engineering.

Methods

The core of this paper lies in the automatic extraction and utilization of syntax features. The authors employ the Stanford Parser (version 1.6.2) to process Mandarin text, obtaining word segmentation, POS tagging, phrase structure trees, and grammatical relations. They then design two sets of syntax features: one for phone duration prediction and another for pause duration prediction.

For phone duration prediction, features are derived from the syntax tree to identify regular accents, which are influenced by syntactic structure (e.g., predicate-object relationships). Features include information about subject-predicate structures, modification structures, object structures, and the syntactic depth of words.

For pause duration prediction, features are based on the analysis of pause distribution across various syntax boundaries. These include features related to subject-predicate boundaries, preposition-object/verb-object boundaries, word compounds, and the presence of specific auxiliary words like "的". The "Pause Level" feature, an empirical assignment to prosodic-word boundaries, is also incorporated.

The duration prediction models themselves are implemented using two types of decision trees: CART models and M5' tree models. The M5' model, capable of handling continuous classes, is expected to yield better results for duration prediction.

Datasets

The study uses a speech corpus of 300 Mandarin sentences read by a native female speaker, collected at the University of Science and Technology of China. This corpus includes pronunciation symbols and prosodic boundary labels. The sentences were pre-processed using the Stanford Chinese Word Segmenter and then parsed by the Factored Chinese version of the Stanford Parser. The corpus was divided into training (90%) and testing (10%) sets.

Key Insights and Ideas

  • Syntax is a valuable, yet underutilized, source for duration prediction: The paper strongly supports the idea that syntactic structure provides crucial information for predicting how long speech segments should last, going beyond traditional phonetic and prosodic features.
  • Automatic extraction is feasible and effective: The success of using features from an automatic parser, despite its known limitations in precision for Chinese, demonstrates a practical path towards integrating complex linguistic information into TTS systems.
  • Syntax features offer complementary information: The improvements seen, particularly for pause and final durations, suggest that syntax features capture aspects of speech timing not fully covered by phonetic and prosodic cues alone.
  • M5' models show an advantage: The M5' regression and model tree approaches consistently outperform CART models, highlighting their suitability for continuous value prediction tasks like duration.

Key Results

The experiments show that incorporating syntax features consistently improves duration prediction accuracy.

  • Phone Duration Prediction: Models using syntax features (e.g., M5p-R-B+S) achieved higher correlation and lower RMSE (Root Mean Squared Error) compared to baseline models. The best performing model, M5p-R-B+S, showed a correlation of 0.97 and an RMSE of 12 ms for initials, and 0.84 and 16 ms for finals.
  • Pause Duration Prediction: Similarly, syntax features led to improved results. The M5p-B+S model achieved the best performance with a correlation of 0.71 and an RMSE of 17 ms.
  • The paper notes that the improvement for initials was less significant, attributing this to potential accent influences being more pronounced on finals and the relatively small training dataset.

Limitations and Future Work

  • Parser Precision: The authors acknowledge that the Chinese syntax parser is not perfect, which limits the precision of the extracted syntax features.
  • Data Sparseness: The relatively small training corpus (270 sentences for modeling) may have contributed to less significant improvements for initials and could be a factor in data sparsity issues.
  • Future Work: The authors plan to explore combining outputs from different Chinese parsers to improve feature quality. They also intend to further investigate the usefulness of syntax information in broader Mandarin prosody modeling.

Summary updated

BibTeX
@inproceedings{wen2010improving,
  author = {Miaomiao Wen and Miaomiao Wang and Keikichi Hirose and Nobuaki Minematsu},
  title = {{Improving Mandarin segmental duration prediction with automatically extracted syntax features}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure