TL;DR
This paper introduces non-linear formant trajectories to model speech dynamics in HMM-based speech synthesis, leading to improved intelligibility and naturalness. By moving beyond simple linear approximations, the research demonstrates a significant step towards more human-like synthetic speech.
What Problem Does This Paper Address?
Generating natural-sounding synthetic speech is a long-standing challenge. Traditional Hidden Markov Model (HMM)-based speech synthesis often models speech dynamics (how speech sounds change over time) using simple linear trajectories. However, these linear models fail to capture the complex, non-linear nature of human articulation, potentially leading to robotic or unnatural-sounding speech. This paper aims to address this limitation by exploring the use of non-linear formant trajectories to better model these dynamics, thereby improving the quality of synthesized speech.
Methods
The core innovation lies in modeling speech dynamics using non-linear formant trajectories. The authors build upon previous work in speech recognition that used non-linear trajectories for phone recognition. For speech synthesis, they employ a standard HMM-based system but train it on 12 Parallel Formant Synthesizer (PFS) control parameters and their time derivatives.
Two types of PFS control parameters are generated to drive the synthesis:
- Piecewise constant PFS control parameters: These are derived using a simplified approach where state durations are estimated from self-transition probabilities and state means are used as output observations.
- Smooth trajectory PFS control parameters: These are generated using a speech parameter generation algorithm (described in [7]) that explicitly considers dynamic features (delta and delta-delta coefficients) and aims to produce non-linear, smooth trajectories. This is the key novel aspect for synthesis.
The HMMs are trained on these 36-dimensional feature vectors (12 parameters + deltas + delta-deltas) at a 10ms frame rate. The synthesized speech is then produced by feeding the generated PFS control parameters into a parallel formant synthesizer.
Datasets
The TIMIT corpus was used for all experiments. Specifically, the male utterances from the TIMIT training set (3,252 utterances, with 8 discarded due to corruption) were used for training the HMMs. The corpus was downsampled to 8KHz to be compatible with the Holmes formant analyzer used to extract the PFS control parameters.
Key Insights and Ideas
- Beyond Linearity: The fundamental insight is that human speech dynamics are inherently non-linear, and simple linear models are insufficient for high-quality synthesis. Capturing these non-linearities is crucial for naturalness.
- Formant Space as an Intermediate Layer: Using formant frequencies (or PFS control parameters, which are closely related) as an intermediate representation in HMMs offers a way to model articulatory dynamics without compromising recognition performance, and as shown here, can improve synthesis.
- Speech Parameter Generation for Smoothness: The speech parameter generation algorithm, previously used for recognition, proves effective in smoothing trajectories and reducing discontinuities, which are common artifacts in HMM synthesis.
- Impact on Different Speech Aspects: While non-linear trajectories improved overall intelligibility and naturalness, their benefit was less pronounced for certain phonetic attributes like voicing and nasality, suggesting that other factors might dominate these specific aspects.
Key Results
Subjective listening tests were conducted to evaluate the synthetic speech:
- Diagnostic Rhyme Test (DRT): Non-linear trajectory-based synthetic speech showed higher intelligibility, with a word accuracy of 70% compared to 67% for piecewise constant parameters.
- Naturalness Test: Non-linear trajectory-based speech scored significantly higher in overall quality (45% vs. 35.9%).
- Statistical Significance: A paired t-test confirmed that the improvements in intelligibility and naturalness due to non-linear trajectories were statistically significant (p < 0.01 for DRT, p < 0.001 for Naturalness).
- Acoustic Attributes: For DRT, non-linear trajectories showed benefits in attributes like "sustension" and "compactness," but not significantly for voicing or nasality.
- Mimicry Test: Speaker adaptation techniques (MLLR, MAP) were explored but proved less successful in the Mimicry test than expected, with results suggesting limitations in adaptation data size or the test methodology for synthesis.
Limitations and Future Work
- Mimicry Test Limitations: The authors acknowledge that the speaker adaptation techniques were not as successful as anticipated in the Mimicry test. They hypothesize that the amount of adaptation data might be too small for speech synthesis compared to speech recognition, or that the TIMIT corpus itself has limitations.
- Overall Quality: While improvements were demonstrated, the authors note that the overall quality of the synthetic speech is still considered "fairly poor" compared to state-of-the-art systems.
- Future Directions:
- Building a TTS system based on multiple-level segmental HMMs (MSHMMs) with a formant-based intermediate layer.
- Generating PFS control parameters directly from these MSHMMs.
- Improving overall synthetic speech quality by incorporating prosodic structure and context-dependent models.
- Further investigation into speaker adaptation techniques for synthesis.
Summary updated