Accent type and phrase boundary estimation using acoustic and language models for automatic prosodic labeling

Tomoki Koriyama, Hiroshi Suzuki, Takashi Nose, Takahiro Shinozaki, Takao KobayashiInterspeech 2014

TL;DR

This paper introduces a novel method for automatically labeling prosodic features like accent type and phrase boundaries in Japanese speech. By combining acoustic and language models, their approach significantly improves accuracy over previous methods, paving the way for more natural and efficient speech synthesis systems.

What Problem Does This Paper Address?

Creating high-quality speech synthesis systems, especially those that mimic natural human speech, relies heavily on accurately labeled prosodic information within speech databases. This includes details like accent type (which affects word meaning in Japanese) and phrase boundaries (pauses or intonational breaks). Manually annotating this prosodic information is a laborious and expensive process, prone to inconsistencies between annotators. Existing automatic methods often oversimplify acoustic features or neglect the crucial role of language models, leading to suboptimal accuracy. This paper aims to bridge this gap by developing a more robust and accurate automatic prosodic labeling technique.

Methods

The core of this paper's contribution lies in its unified framework for simultaneously estimating accent type and phrase boundary sequences. They propose using Conditional Random Fields (CRFs) for the language model, which leverages linguistic features like part-of-speech (POS) tags and word information. For the acoustic model, they employ Hidden Markov Models (HMMs), which are adept at capturing the continuous nature of acoustic features like fundamental frequency (F0) contours.

The process involves:

  1. Language Model Prediction: Using CRFs, they first predict N1-best candidates for accent phrase boundary sequences and then N2-best candidates for accent type sequences, conditioned on the predicted boundaries and linguistic features.
  2. Acoustic Model Scoring: For each combined accent type and phrase boundary candidate sequence, they calculate the probability of the observed acoustic features using HMMs.
  3. Joint Optimization: Finally, they combine the probabilities from the language and acoustic models to select the optimal sequence of accent types and phrase boundaries. This joint approach acknowledges that prosodic features are interdependent.

A key innovation is the use of HMMs for acoustic modeling, which is presented as a significant improvement over previous methods that relied on simplified polygonal approximations of F0 contours.

Datasets

The experiments were conducted using the ATR Japanese speech database set B, which includes speech data from six male speakers. For evaluation, 53 sentences from one target speaker were used. For model training, 450 sentences from the same target speaker were used in the "speaker-closed" case, and 2250 utterances from five other male speakers were used in the "speaker-open" case.

Key Insights and Ideas

  • Unified Framework is Crucial: The paper strongly advocates for simultaneously estimating accent types and phrase boundaries, as these prosodic elements are inherently linked. Treating them independently can lead to suboptimal results.
  • HMMs for Continuous Prosody: Employing HMMs for acoustic modeling is a significant step forward, as they can better capture the dynamic and continuous nature of F0 contours compared to simpler approximation methods. This leads to more accurate prosodic feature representation.
  • Language Models Matter: The success of their approach hinges on the effective integration of language models (CRFs) alongside acoustic models. This combination allows the system to leverage both linguistic context and acoustic evidence for more informed predictions.
  • N-best Candidates for Practicality: Due to the combinatorial explosion of possible prosodic sequences, the authors use an N-best candidate approach to manage computational complexity while still exploring a sufficient range of possibilities.

Key Results

  • Improved Accuracy: The proposed method demonstrated higher accuracy for both accent type estimation (6.5-7.3% higher than conventional methods in speaker-closed and speaker-open cases, respectively) and accent phrase boundary estimation (achieving F-scores of 0.949 and 0.932 in speaker-closed and speaker-open cases, respectively).
  • Weighting is Key: The performance of the system is sensitive to the weight (α) that balances the influence of acoustic and language models. The optimal weights were found to be around 2.5 × 10⁻² for accent type and 10⁻² for phrase boundary estimation.
  • Coverage of Candidates: The study found that using 10-best candidates for phrase boundaries (N₁) and 50-best for accent types (N₂) provided good coverage of the correct sequences.

Limitations and Future Work

  • Weight Optimization: The authors acknowledge that the performance is dependent on the chosen weight (α) that balances acoustic and language model contributions. Future work should focus on methods for automatically determining this optimal weight.
  • Data and Model Refinement: They suggest utilizing more data for language model training and refining the acoustic model to further enhance accuracy.
  • Application to TTS: A crucial next step is to assess the effectiveness of the automatically annotated data for improving statistical speech synthesis systems.
  • Speaker Dependency: While speaker-open experiments were conducted, further research into reducing speaker dependency remains a valuable direction.

Summary updated

BibTeX
@inproceedings{koriyama2014accent,
  author = {Tomoki Koriyama and Hiroshi Suzuki and Takashi Nose and Takahiro Shinozaki and Takao Kobayashi},
  title = {{Accent type and phrase boundary estimation using acoustic and language models for automatic prosodic labeling}},
  booktitle = {Interspeech},
  year = {2014},
}
Figure