TL;DR
This paper investigates how crucial linguistic features like part-of-speech tags, phrase boundaries, and pitch accents impact the naturalness of HMM-based speech synthesis. By systematically removing these features during training and evaluation, the authors demonstrate their significant contribution to synthetic speech quality and explore the implications of noisy annotations.
What Problem Does This Paper Address?
Building natural-sounding synthetic speech, particularly with data-driven methods like HMM-based Text-to-Speech (TTS), relies on a rich linguistic specification that bridges the gap between text and audio. This specification includes not only basic phonetic information but also higher-level linguistic cues such as syntactic categories (part-of-speech), intonational phrase boundaries, pitch accents, and boundary tones. However, obtaining accurate, hand-annotated data for these higher-level features is expensive and time-consuming, especially for languages lacking extensive linguistic resources. This paper aims to quantify the contribution of these higher-level features to synthetic speech naturalness, providing a baseline for future research focused on using "shallow" features that are easier to extract from text. Additionally, it examines the impact of imperfect, automatically generated annotations on system performance.
Methods
The core of this research involves building and evaluating multiple HMM-based speech synthesis systems trained on different subsets of linguistic features. The authors used the Boston University Radio News Corpus (BURN), specifically focusing on data with high-quality, manually annotated ToBI (Tones and Break Indices) labels.
They created two sets of utterance structures:
- Gold: Utilizes manually annotated or verified higher-level features (part-of-speech, phrase boundaries, pitch accents, boundary tones).
- Auto: Relies on automatically predicted higher-level features, using Festival's pre-trained models.
Lexical features (phonemes, stress, word boundaries) were derived similarly for both sets. Twelve HTS (HMM-based Text-to-Speech Synthesis System) systems were built, systematically omitting questions related to specific linguistic feature sets during decision-tree based context clustering. These systems were then evaluated through a subjective AB test where listeners compared pairs of synthetic utterances.
Datasets
- Training and Test Data: Boston University Radio News Corpus (BURN) [4]. The speaker 'f2b' was used for training, and speakers 'mlb' and 'm2b' for testing.
- Annotation: The BURN corpus provided manually assigned ToBI labels. For the "Auto" systems, Festival's pre-trained models were used for automatic annotation.
- Lexicon: Festival's copy of the CMU Pronouncing Dictionary [6] was used, augmented with manually transcribed out-of-lexicon words.
Key Insights and Ideas
- Higher-level features matter: Removing higher-level linguistic features significantly degrades the naturalness of synthetic speech, indicating their crucial role in prosody and intelligibility.
- Manual vs. Automatic Annotation: Systems trained with manually annotated "Gold" labels generally perform better than those trained with automatically generated "Auto" labels, suggesting that annotation quality is paramount.
- Surrogacy Effects: When higher-level features are removed, the synthesis system compensates by relying more heavily on lower-level features (e.g., quinphones) or by finding patterns in the remaining features that mimic the missing ones. For instance, removing pitch accent and tone features leads to increased use of syllable-level phrase boundary questions.
- Feature Importance Hierarchy: The study suggests that pitch accent, boundary tone, and part-of-speech features contribute more to listener preference than phrase features alone.
- Impact of Noise: The authors hypothesize that the impact of noisy automatic annotations might be less detrimental than the complete removal of features, especially in systems that already rely on automatic prediction.
Key Results
- Subjective Evaluation (Figure 1): While only one comparison (G1 vs. G4) was statistically significant, the overall trend showed a preference for systems with more linguistic features included. G1 (full features, Gold labels) was generally preferred over systems with fewer features.
- Question Distribution (Figure 2): The distribution of questions asked during synthesis varied significantly based on the linguistic features available during training. Removing features led to increased reliance on others, demonstrating "surrogacy" effects. For example, removing Tones and Accents (T1-T4) led to a sharp increase in questions about phrase boundaries (P1).
Limitations and Future Work
- Dataset Size: The training data (55 minutes) was minimal for a speaker-dependent system and not phonetically balanced, which may have limited the segmental quality of the synthesized voices.
- Statistical Significance: Many of the observed trends in the subjective evaluation were not statistically significant, likely due to the small number of listeners and utterances. The authors suggest that a more extensive evaluation would be needed to confirm these trends.
- Future Work: The authors plan to investigate the impact of noisy automatic annotations further and explore the use of "shallow" features that are easier to extract from text, aiming to replace the expensive higher-level features without significant loss in quality. They also intend to analyze the "Auto" systems (A1-A4) more thoroughly to understand the impact of noisy labels on listener preference.
Summary updated