A method of multi-layered speech segmentation tailored for speech synthesis

Takashi SaitoInterspeech 2005

TL;DR

This paper introduces a novel multi-layered speech segmentation method specifically designed for speech synthesis. By extracting not just phonemes but also breath groups and accent phrases, the method aims to capture the nuances of prosody and acoustics necessary for realistic synthesized speech, demonstrating robust performance across various speakers and speaking styles.

What Problem Does This Paper Address?

Creating high-quality, personalized speech synthesis voices is a labor-intensive process, largely due to the need for detailed acoustic and prosodic labeling of speech corpora. While phoneme-level segmentation is a standard component, it's insufficient for capturing the natural flow and expressiveness of human speech. Synthesizers require richer information, including breath groups (natural pauses for breathing), accent phrases (units of prosodic meaning), and pitch contours, to accurately reproduce a speaker's unique characteristics. Existing methods often rely on complex training procedures or are not optimized for the specific needs of speech synthesis, posing a barrier to widespread adoption. This paper addresses the need for an automated, robust, and multi-layered segmentation approach that directly supports speech synthesis voice creation.

Methods

The proposed method employs a multi-layered approach, integrating information from text analysis, pitch-mark detection, and a novel optimization loop.

  • Text Analysis: A text-to-speech (TTS) engine's analysis module is used to generate phonetic transcriptions and identify accentual phrases from input text. This module also predicts phoneme durations to help determine pause positions.
  • Pitch-Mark Detection: A wavelet-based method extracts pitch-mark information, which is crucial for determining voiced/voiceless segments and is used to derive a "segmental reliability" metric.
  • Segmental Optimization Loop: This is the core of the method.
  • Breath-Group Segmentation: The algorithm first identifies intra-sentence pauses based on log power and spectrum analysis, using a threshold (Lp) for minimum pause length. Accentual phrase boundaries are then used to decompose the utterance into breath groups.
  • Phoneme Segmentation: Within each breath group, phonemic alignment is performed using a dynamic time warping (DTW) approach, leveraging a reference template from a similar TTS engine. This DTW-based alignment is chosen for its simplicity and to avoid complex training procedures.
  • Segmental Reliability: A key innovation is the introduction of "segmental reliability" (Rs). This metric quantifies the consistency between the transcribed phoneme properties (voiced/voiceless) and the actual acoustic features derived from pitch marks. Rs is calculated based on several predefined check items, penalizing inconsistencies. The segmentation process is optimized to maximize Rs.
  • Phrase Segmentation: Finally, the segmented phonemes are grouped into accentual phrases.

Datasets

The authors evaluated their method using a script set of 503 sentences from the ATR continuous speech database. They prepared 6 speech corpora from this script, uttered by professional speakers with varying speaking styles and characteristics:

  • S1F, S2F, S3M: Spoken at a relatively slow speed.
  • N1F: Spoken at a normal speed.
  • C1F, C2F: Spoken in character voices.

Key Insights and Ideas

  • Multi-layered segmentation is crucial for synthesis: Simply segmenting phonemes isn't enough; capturing breath groups and accent phrases is essential for realistic prosody.
  • Leveraging TTS engines: Using a TTS engine's analysis module and reference templates simplifies the process and avoids the need for extensive training data for the segmentation tool itself.
  • Segmental reliability as an optimization metric: The novel Rs metric effectively guides the phoneme segmentation by ensuring consistency between phonetic transcription and acoustic evidence, leading to more robust results.
  • DTW for phonemic alignment: A simple DTW approach is sufficient and practical for phonemic alignment, especially when a reference template from a similar system is available, avoiding the complexity of HMM-based methods.
  • Robustness to speaker and style variations: The method demonstrates remarkable stability across different speakers and speaking styles, a critical factor for creating diverse voice inventories.

Key Results

  • Phoneme Segmentation: The optimized segmentation achieved average errors of 9.1 to 13.5 ms across corpora, with a significant reduction in error compared to using a fixed Lp threshold (14.0% to 33.5% improvement). The method showed good accuracy even for character voices, which differ significantly from standard speech.
  • Accentual Phrase Segmentation: Over 90% of accentual phrase boundaries were correctly segmented at the intermediate level. At the final level, accuracy decreased slightly (by 4-7%) due to short pauses, but this did not significantly impact FO unit extraction.
  • Accent Type Setting: The accuracy of accent type setting for correctly segmented phrases was high, ranging from 86.4% to 93.1%.
  • Efficiency: The paper notes that preliminary feasibility studies showed a reduction in voice inventory creation time from over three months to about one month using this method.

Limitations and Future Work

  • Short Pauses: The accuracy of accentual phrase segmentation at the final level is affected by intra-sentence pauses shorter than the minimum threshold (Lp) used for breath-group segmentation.
  • Accent Type Setting Accuracy: While good, the accent type setting accuracy is partly dependent on the text analysis module and potential mismatches between the script and actual utterance.
  • Future Work: The authors suggest that the method's robustness and efficiency make it highly suitable for generic speech database construction, benefiting both speech synthesis and general speech corpus research. The reduction in creation time for voice inventories is a significant practical outcome.

Summary updated

BibTeX
@inproceedings{saito2005method,
  author = {Takashi Saito},
  title = {{A method of multi-layered speech segmentation tailored for speech synthesis}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure