A hybrid TTS approach for prosody and acoustic modules

Iñaki Sainz, Daniel Erro, Eva Navas, Inma HernáezInterspeech 2011

TL;DR

This paper presents a novel hybrid Text-to-Speech (TTS) system that combines the strengths of unit selection (US) and statistical parametric (SP) synthesis. By fusing prosody and acoustic modules from both approaches, the system achieves more natural and consistent speech than either method alone, particularly in broader domains.

What Problem Does This Paper Address?

Traditional Text-to-Speech (TTS) systems fall into two main categories: unit selection (US) and statistical parametric (SP) synthesis. US systems excel at producing natural-sounding speech by concatenating pre-recorded speech units, but they suffer from variable quality and large footprint. SP systems, on the other hand, offer consistent quality and a smaller footprint by modeling speech parameters, but often produce less natural, "buzzy" speech due to vocoding. Existing hybrid approaches have primarily focused on using SP for prosody generation or acoustic parameter generation, but this paper tackles the challenge of integrating both prosody and acoustic modules from both US and SP systems to achieve a more robust and natural synthesis.

Methods

The proposed hybrid TTS system integrates two baseline systems: the AhoTTS unit selection system and an HTS-based statistical parametric system. The core innovation lies in the fusion of these systems at both the prosody and acoustic module levels.

  • Prosody Module Fusion:
    • Duration Prediction: A linear combination of duration predictions from the HTS system and the AhoTTS prosody module (using Random Forests and CART) is employed. This fusion is bidirectional, meaning the output feeds both the HTS parameter generator and the standard prosody module.
  • Intonation Contour Prediction: A multi-stage process is used to fuse intonation curves from HTS and the AhoTTS prosody module. This involves interpolating f0 values, segmenting at the phone level, performing 3-point pitch stylization for voiced phonemes, and finally, a weighted linear combination of aligned phone-sized pitch portions.
  • Acoustic Module Fusion:
    • Unlike many hybrid systems that rely solely on SP acoustic trajectories, this approach maintains the standard linguistic and prosodic target sub-costs from the US system.
  • A new sub-cost, Spectral Distance, is introduced. This is a frame-based Euclidean distance between the target (HTS output) and candidate units, aligned using Dynamic Time Warping (DTW). This spectral distance is weighted based on phonetic classes (vowels, voiced, unvoiced consonants). This aims to prevent the selection of "bad units" and improve synthesis consistency.

Datasets

  • Training: A two-hour corpus of male voice recordings in Spanish, recorded at 16 kHz, was used. This corpus was provided by the Albayzin 2010 TTS evaluation campaign. Additional material included automatic segmentation and Intonation Break (IB) labels.
  • Evaluation: 350 natural sentences, not used during voice building, were provided by the Albayzin 2010 TTS evaluation campaign organizers. These were used for objective and subjective evaluations.

Key Insights and Ideas

  • Fusion at Multiple Levels is Key: The paper argues that combining both prosody and acoustic modules from US and SP systems leads to a more robust and natural synthesis than focusing on just one aspect.
  • Prosody Fusion Improves Consistency: Fusing duration and intonation predictions from different methods (HTS and AhoTTS prosody module) leads to more consistent prosody generation, even if individual objective measures don't always show dramatic improvements.
  • Spectral Distance Prevents "Bad Units": The introduction of a spectral distance sub-cost in the unit selection process is crucial for preventing the selection of poorly formed or mislabeled units, thereby improving overall synthesis quality.
  • Intonation Break (IB) Information is Valuable: The inclusion of IB information significantly improves both duration and intonation prediction accuracy, highlighting its importance in modeling prosody.
  • Subjective Evaluation is Crucial: While objective measures provide insights, subjective evaluations (like CMOS tests) are essential for truly assessing the perceptual quality and naturalness of TTS systems.

Key Results

  • Objective Evaluation (Duration Prediction): The hybrid method improved upon the CART+RF method alone, and the inclusion of IB information further boosted prediction accuracy across all methods.
  • Objective Evaluation (Intonation Contour Prediction): The hybrid method slightly improved HTS prediction, but differences were not statistically significant. IB information also played a significant role here.
  • Subjective Evaluation (CMOS Test): In a randomized preference test, the hybrid system was preferred over both the HTS-based system (0.63 CMOS) and the standard US system (0.31 CMOS), with statistical significance.
  • Albayzin 2010 TTS Evaluation: The hybrid system performed well in a large-scale evaluation, achieving high scores for similarity to the original voice (4.07 MOS), naturalness (3.71 MOS), and intelligibility (17% WER). It was significantly better than other synthetic systems in naturalness.
  • Loudspeaker vs. Headphones: Using loudspeakers for evaluation reduced the perceived gap in naturalness between the hybrid system and the HTS-based system, suggesting that headphone limitations might influence perception.

Limitations and Future Work

  • Gap with Natural Voice: Despite improvements, a considerable gap still exists between synthetic and natural voices, as indicated by the Albayzin 2010 evaluation results where all synthetic systems performed worse than the natural voice.
  • Manual Tuning of Weights: The weights for the linear combination in the prosody fusion process were manually tuned. Future work could explore automatic methods for determining these weights.
  • Computational Cost: The spectral distance calculation in the acoustic module is computationally intensive. The paper mentions using it in a pre-selection stage to mitigate this, but further optimization might be beneficial.
  • Further Exploration of Fusion Techniques: The paper suggests that exploring more advanced fusion techniques for both prosody and acoustic modules could lead to further improvements.

Summary updated

BibTeX
@inproceedings{sainz2011hybrid,
  author = {Iñaki Sainz and Daniel Erro and Eva Navas and Inma Hernáez},
  title = {{A hybrid TTS approach for prosody and acoustic modules}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure