TL;DR
This paper introduces a novel "two-pass" decision tree construction method for HMM-based speech synthesis. This technique allows for unsupervised speaker adaptation without needing separate acoustic models for recognition or complex linguistic analysis of the transcribed data, achieving performance comparable to supervised methods.
What Problem Does This Paper Address?
Hidden Markov Model (HMM)-based speech synthesis offers advantages over traditional concatenative methods, particularly in speaker adaptation. However, adapting these models to new speakers using unsupervised data (audio without pre-existing transcriptions) presents significant challenges. The primary hurdles are:
- Model Mismatch: HMM synthesis models use "full contexts" (including phonetic and suprasegmental information like stress) that are too complex for standard Automatic Speech Recognition (ASR) systems to efficiently transcribe. This typically necessitates using a separate set of simpler ASR-focused acoustic models.
- Linguistic Analysis Overhead: Even if an initial transcription is obtained, it's usually at the word level. Converting this to the detailed phonetic and contextual information needed for HMM synthesis requires linguistic analysis, which can introduce and amplify errors from the initial transcription.
This paper aims to overcome these limitations, enabling robust unsupervised adaptation of HMM synthesis models without relying on supplementary ASR models or intricate linguistic processing.
Methods
The core innovation is a two-pass decision tree construction method.
- Pass 1: Builds a phonetic decision tree using only basic phonetic questions (left, right, central phonemes). This tree generates "tied triphone contexts" suitable for ASR but doesn't estimate acoustic model parameters yet.
- Pass 2: Extends the Pass 1 tree by adding questions about suprasegmental information (e.g., syllable count, stress). This creates an "extended decision tree" that defines "tied full contexts" for synthesis. Crucially, each leaf node in Pass 2 has an ancestor in Pass 1, creating a direct link.
This "two-pass" structure allows for a mapping between the synthesis-oriented full context models and the ASR-oriented triphone models. This mapping is key to the unsupervised adaptation process:
- Transcription: The triphone models (derived from the Pass 1 tree) are used to transcribe the unlabeled adaptation data.
- Adaptation: Two methods are proposed:
- Triphone Adaptation: Adapts the triphone models using the transcribed data, then maps them back to full context models.
- Full Adaptation: Uses linguistic analysis to derive full context labels from the transcribed data and adapts the full context models directly.
The paper hypothesizes that triphone adaptation might be more robust due to avoiding potential errors from linguistic analysis.
Datasets
- Training: Wall Street Journal (WSJ) SI84 dataset was used to estimate average voice models.
- Adaptation Data: Speaker 440M from the 1993 ARPA evaluation dataset was used, with varying amounts (10 and 40 utterances).
- Test Data: A subset of the 1992 ARPA speaker-independent read 5k test dataset was used for evaluation.
- Language Model: A 20k bigram language model from the WSJ corpus was used for the recognition step.
No new benchmarks were introduced; standard evaluation datasets were utilized.
Key Insights and Ideas
- Unified Model Framework: The two-pass decision tree construction elegantly bridges the gap between ASR and TTS models, allowing a single set of components to serve both purposes. This is a significant simplification.
- Unsupervised Adaptation Without Pain: The method effectively bypasses the need for supplementary ASR models and complex linguistic analysis, which were previously considered essential for unsupervised adaptation. This drastically reduces the complexity and potential error sources.
- Linguistic Analysis is Optional (and potentially detrimental): The experiments suggest that linguistic analysis in the unsupervised adaptation pipeline doesn't necessarily improve results and might even introduce errors. This is a crucial finding for simplifying future adaptation systems.
- Performance Parity: The proposed unsupervised method achieves performance indistinguishable from supervised adaptation, a major step forward for practical applications.
Key Results
- Similarity to Target Speaker: Unsupervised adaptation using the two-pass method (systems H and J) achieved results statistically indistinguishable from supervised adaptation (systems G and I) and significantly better than the average voice model (B). This demonstrates that the core goal of unsupervised adaptation was met.
- Naturalness: The two-pass decision tree construction (System B) did not compromise the naturalness of the synthesized speech compared to standard decision tree construction (System A). This confirms that the novel method doesn't degrade synthesis quality.
- Linguistic Analysis Impact: No significant difference was observed between "Full Adaptation" (system H) and "Triphone Adaptation" (system J) in the unsupervised case, suggesting linguistic analysis is not critical for performance.
Limitations and Future Work
- Limitations:
- The paper notes that the volume of adaptation data (10 or 40 utterances) was insufficient to significantly improve the naturalness of the synthesized speech, although similarity to the target speaker was improved.
- The unsupervised transcription achieved a phoneme error rate of 47.1% (word error rate 72.5%), indicating room for improvement in the recognition accuracy itself.
- Future Work: Not explicitly detailed, but the findings suggest further exploration into optimizing the unsupervised transcription step and investigating the impact of larger adaptation datasets on naturalness would be valuable. The paper also implies that the efficiency gains from omitting linguistic analysis could be further leveraged.
Summary updated