Fully automatic segmentation for prosodic speech corpora

Sarah Hoffmann, Beat PfisterInterspeech 2010

TL;DR

This paper introduces a fully automatic method for segmenting speech into phonemes, crucial for building prosody models in speech synthesis systems. By combining an improved HMM-based segmentation with a novel boundary correction technique that leverages signal properties, they achieve high accuracy without relying on any manually labeled data, making it applicable to a wider range of languages.

What Problem Does This Paper Address?

Accurately segmenting speech into its constituent phonemes is a fundamental step in many speech processing tasks, particularly for training prosody models in text-to-speech (TTS) systems. Traditional methods often require a significant amount of manually annotated speech data for training or post-processing, which is time-consuming and limits their applicability to less common languages. The challenge is to develop a method that can achieve high segmentation accuracy automatically, using only the speech corpus and its automatically generated transcription, even in the presence of transcription errors.

Methods

The proposed method consists of a two-stage process:

  1. Initial HMM-based Segmentation with Boundary Correction:
    • Automatic Transcription: An initial transcription is generated from the orthographic text using a TTS system's syntactic component. This transcription may contain errors.
  2. Flat-Start HMM Training: Phoneme models are initialized without prior knowledge and trained iteratively using embedded training on the corpus.
  3. Forced Alignment: A preliminary segmentation is produced using these HMMs.
  4. Silence, Plosive, and Glottal Stop Insertion: Special handling is included for these non-speech sounds, which are often absent in automatic transcriptions but crucial for prosody modeling. Silence and glottal stop models are learned, and plosive pauses are inserted before plosives.
  5. Boundary Correction: A novel algorithm corrects the boundaries identified in the preliminary segmentation. This correction leverages the local feature distance between adjacent phones, identifying frames that are closer to the "core" of one phone than the other. This step aims to refine boundaries without manual intervention.
  6. Second Stage Refinement:
    • Isolated-Unit Training: Phone models are re-estimated using only the preliminary segmentation from the first stage. This helps preserve boundary information.
  7. Second Forced Alignment and Correction: Another segmentation is performed, and the boundary correction is applied again. This stage can be iterated to further improve accuracy.

The key novelty lies in the boundary correction algorithm, which uses signal properties (feature distances) to refine HMM-based segmentations, and its integration into a fully automatic pipeline that avoids manual annotation.

Datasets

  • German Prosody Corpus: Used for evaluation. Consists of 186 sentences spoken by a single male speaker, with a pre-existing manual segmentation.
  • TIMIT Speech Corpus: Used for evaluation. A larger, multi-dialect, multi-speaker corpus with both manual and automatic transcriptions available.

Key Insights and Ideas

  • The Trade-off in HMM Training: Iterative embedded training of HMMs on a corpus, while improving model coverage, can actually decrease boundary accuracy by incorporating parts of adjacent phoneme contexts into the models.
  • Signal Properties for Boundary Refinement: The local feature distance between adjacent phones provides a robust signal for identifying accurate phone boundaries, even when HMMs might be slightly off.
  • Fully Automatic is Achievable: The proposed method demonstrates that high-quality phonetic segmentation can be achieved without any manual labeling, significantly reducing the barrier to entry for new languages.
  • Importance of Non-Speech Sounds: Explicitly modeling and segmenting silences, plosive pauses, and glottal stops is crucial for accurate prosody modeling and improves the overall segmentation process.

Key Results

  • German Prosody Corpus: The boundary correction significantly improved segmentation accuracy. For a 20ms deviation, the accuracy increased from 86.61% after initial HMM training to 89.14% after boundary correction in the second stage, and further to 89.45% after 10 iterations. The precision (5ms deviation) improved from 40.23% to 53.03%.
  • TIMIT Corpus: Similar improvements were observed. For manual transcription, boundary correction improved accuracy to 90.23% (20ms deviation) and precision to 54.26% (5ms deviation). For automatic transcription, accuracy reached 88.40% (20ms deviation) and precision 53.48% (5ms deviation).
  • Robustness to Transcription Errors: The method showed good robustness when evaluated with automatic transcriptions, with only a slight decrease in performance compared to manual transcriptions.

Limitations and Future Work

  • Comparison to State-of-the-Art: While the method achieves high accuracy without manual data, the authors acknowledge that some statistical boundary correction methods (e.g., using neural networks or regression trees) might achieve higher total precision, but these typically require manual data for training.
  • Future Work: The paper suggests that introducing more variants into the HMMs could further counter the effects of transcription errors, especially in multi-speaker corpora. Further exploration of perceptual features for boundary correction is also implied. The authors also note that the desirability of higher agreement with reference labels depends on the specific downstream application.

Summary updated

BibTeX
@inproceedings{hoffmann2010fully,
  author = {Sarah Hoffmann and Beat Pfister},
  title = {{Fully automatic segmentation for prosodic speech corpora}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure