An HMM Trajectory Tiling (HTT) Approach to High Quality TTS - Microsoft Entry to Blizzard Challenge 2010

Yao Qian, Zhi-Jie Yan, Yi-Jian Wu, Frank K. Soong, Guoliang Zhang, Lijuan WangBlizzard Challenge 2010 · OpenReview

TL;DR

This paper introduces the HMM Trajectory Tiling (HTT) approach for high-quality text-to-speech (TTS) synthesis. HTT refines HMMs using Minimum Generation Error (MGE) to generate precise acoustic trajectories, which then guide a search for the best matching waveform segments, resulting in natural and intelligible synthesized speech.

What Problem Does This Paper Address?

Traditional TTS systems fall into two main camps: unit selection (which can sound very natural but may have glitches) and HMM-based parametric synthesis (which is smooth but can have a "vocoded" or muffled quality). Hybrid approaches aim to combine the strengths of both. However, a key challenge remains in effectively guiding the selection of natural waveform units using the smoother, but less perceptually accurate, trajectories generated by HMMs. This paper tackles the problem of generating highly natural and intelligible synthesized speech by improving the alignment between HMM-generated acoustic targets and the actual waveform units used for concatenation.

Methods

The core of the HTT approach lies in its two-stage refinement process:

  1. Refined HMM Training: Standard HMMs are first trained using the Maximum Likelihood (ML) criterion. Then, they are further optimized using the Minimum Generation Error (MGE) criterion. MGE aims to minimize the discrepancy between the HMM-generated parameter trajectories and the actual trajectories from the training data. This process also refines state alignments and improves voiced/unvoiced (v/u) generation for pitch.
  2. HMM Trajectory Tiling (HTT) Synthesis:
    • Trajectory Generation: For a given text, the refined HMM generates a precise acoustic parameter trajectory (including spectral envelope, fundamental frequency, and duration).
  3. Unit Sausage Construction: This trajectory guides the selection of potential waveform units from a database. A "unit sausage" (or lattice) is created, containing candidate units that closely match the HMM trajectory. Distances are calculated based on log-domain differences in F0 and gain, and a weighted RMS distance for Line Spectrum Pairs (LSPs).
  4. Waveform Unit Selection: Within the unit sausage, a search is performed to find the optimal sequence of waveform units. This search is guided by Normalized Cross-Correlation (NCC), a measure that is effective at preserving spectral similarity, phase continuity, and identifying the best concatenation time instants. Dynamic programming (Viterbi algorithm) is used to find the path that maximizes the accumulated NCC.
  5. Concatenation: Adjacent waveform units along the optimal path are then shifted by the best offset and concatenated, often with triangular cross-fading.

The paper also mentions flexible unit selection, allowing for half-phone, state, or phone units depending on the corpus size to balance quality and availability.

Datasets

The system was evaluated on four tasks from the Blizzard Challenge 2010:

  • EH1: 5 hours of RP-accented male speech (Phonetic Arts).
  • EH2: 1 hour of RP-accented male speech (University of Edinburgh).
  • MH1: 9 hours of Mandarin Chinese speech (Chinese Academy of Sciences).
  • MH2: 1 hour of Mandarin Chinese speech (Chinese Academy of Sciences).

The paper notes that for the Mandarin tasks, they used their own automatic labeling system, which differs from system C.

Key Insights and Ideas

  • MGE for Trajectory Precision: The key insight is that refining HMMs with MGE significantly improves the accuracy and perceptual relevance of the generated acoustic trajectories. This provides a much better "guide" for waveform selection.
  • NCC as a Powerful Concatenation Metric: Normalized Cross-Correlation is highlighted as a superior metric for selecting the best waveform segments for concatenation. It effectively balances spectral similarity, phase continuity, and precise alignment, leading to smoother transitions.
  • Hybrid Synergy: The HTT approach demonstrates how combining the strengths of HMMs (smoothness, precise guidance) and unit selection (naturalness, perceptual fidelity) can yield superior results compared to either method alone.
  • Flexibility in Unit Granularity: The ability to adapt the unit size (half-phone, state, phone) based on corpus size is crucial for practical implementation and maintaining quality across different datasets.

Key Results

The HTT approach, represented by letter 'J', performed very well in the Blizzard Challenge 2010:

  • EH1 Task: Achieved the second-best scores for similarity and naturalness, and the best score for intelligibility (15% WER, slightly better than the second-best system R). It was statistically indistinguishable from natural speech.
  • MH1 Task: Achieved the best similarity score (3.9) and tied for first place in naturalness (3.9). Intelligibility (CER) was 22%, slightly worse than the best system C (21%), but the authors note differences in labeling systems.
  • EH2 and MH2 Tasks (smaller databases): The system consistently ranked within the top 3 for all evaluation metrics.

Qualitatively, the synthesized speech was described as "high quality, natural sounding... without sacrificing high intelligibility."

Limitations and Future Work

  • Labeling System Differences: For the Mandarin tasks, the use of an automatic labeling system that differs from other participants makes direct comparison with system C challenging.
  • Corpus Size and Unit Granularity: While the paper discusses adapting unit granularity based on corpus size, the optimal choice for different corpus sizes and languages might require further investigation.
  • Future Work: The paper doesn't explicitly detail future work, but the success of the HTT approach suggests further exploration into MGE training, NCC-based search, and potentially more sophisticated unit selection strategies could be beneficial. The inherent limitations of HMMs (e.g., the source-filter assumption) might still be a ceiling for ultimate naturalness, hinting at directions for future research beyond this specific hybrid model.

Summary updated

Abstract

We propose an HMM Trajectory Tiling (HTT) approach to high quality TTS, which is our entry to Blizzard Challenge 2010. In HTT, first refined HMM is trained with the Minimum Generation Error (MGE) criterion; then trajectory generated by the refined HMM is to guide the search for finding the closest waveform segment "tiles" in synthesis. Normalized distances between HMM trajectory and those of the waveform unit candidates are used for selecting final candidates in a unit sausage (lattice). Normalized cross-correlation, a good concatenation measure for its high relevance to spectral similarity, phase continuity and concatenation time instants, is used for finding the best unit sequence in the sausage. The sequence serves as the best segment tiles to closely follow the HMM trajectory guide. Tested in four tasks, {EH1, EH2, MH1 and MH2}, of Blizzard Challenge 2010, the new HTT approach delivers high quality, natural sounding TTS speech without sacrificing high intelligibility. Subjectively, they are confirmed by naturalness and intelligibility listening test scores.
BibTeX
@inproceedings{qian2010hmm,
  author = {Yao Qian and Zhi-Jie Yan and Yi-Jian Wu and Frank K. Soong and Guoliang Zhang and Lijuan Wang},
  title = {{An HMM Trajectory Tiling (HTT) Approach to High Quality TTS - Microsoft Entry to Blizzard Challenge 2010}},
  booktitle = {Blizzard Challenge},
  year = {2010},
  url = {https://openreview.net/forum?id=ZsR5EuTPuW},
}
Figure