TL;DR
This paper introduces a refined approach to training acoustic models for speech synthesis, combining linear transformation-based adaptation with ex-post Maximum A Posteriori (MAP) estimation. This method significantly improves the naturalness of synthesized speech, especially when large amounts of speaker data are available, outperforming traditional speaker-dependent models.
What Problem Does This Paper Address?
Creating speech synthesis systems that can mimic any speaker's voice (arbitrary speaker synthesis) is a challenging goal. A common approach involves adapting a general "average voice" model to a specific target speaker. While linear transformation methods have shown promise, they can lead to model errors when substantial speaker-specific data is available, potentially resulting in less natural speech compared to models trained solely on that speaker's data (speaker-dependent or SD models). This paper addresses this limitation by proposing a combined approach that leverages the strengths of both linear transformation and MAP estimation to achieve more natural and accurate synthesized speech, even with abundant target speaker data.
Methods
The core of this paper's contribution lies in a two-stage adaptation process for Hidden Semi-Markov Model (HSMM)-based speech synthesis.
- Linear Transformation-based Adaptation: This is a standard technique where the average voice model's parameters are adjusted using a linear transformation to better fit the target speaker's data. The paper uses a structural maximum a posteriori linear regression (SMAPLR) method for this.
- Ex-post MAP Estimation: This is the novel addition. After the initial linear transformation, an ex-post MAP estimation is applied. This process essentially refines the adapted model by incorporating prior knowledge (from the average voice model) and the adaptation data in a Bayesian framework. The authors show that this MAP estimation asymptotically approaches Maximum Likelihood (ML) estimation (similar to SD model training) as more adaptation data becomes available, but crucially, it does so while respecting the tying topology of the adapted average voice model.
The paper also investigates the impact of the "tying topology" – how model parameters are shared across different states and speakers – on the adaptation performance.
Datasets
The experiments utilized speech data from multiple speakers:
- Training Data: Six male and five female speakers' speech samples were used. This included utterances from the ATR Japanese speech database (Set B) and neutral reading speech.
- Evaluation Data: Fifty test sentences, not used in training or adaptation, were employed for evaluation.
- Adaptation Data: For experiments, a set of 450 sentences from the target speaker was used for adaptation.
The paper does not introduce new benchmarks but evaluates its method against existing speaker-dependent (SD) models.
Key Insights and Ideas
- MAP Refinement Bridges the Gap: The ex-post MAP estimation effectively bridges the performance gap between adapted average voice models and speaker-dependent models when large amounts of target speaker data are available.
- Asymptotic Convergence: The MAP estimation, when applied to the linearly transformed average voice model, asymptotically converges to the ML estimation (akin to SD model training) as the amount of adaptation data increases, provided the tying topology is the same.
- Tying Topology Matters: The complexity of the tying topology, influenced by the amount of data used to train the average voice model, plays a significant role. Larger average voice models with more complex tying topologies can capture more speaker-specific information, leading to better synthesis quality.
- Average Voice Model Advantage: With sufficient training data for the average voice model, the proposed adapted average voice model can generate more natural-sounding speech than a speaker-dependent model trained on the same amount of target speaker data.
Key Results
- Objective Evaluation: The SMAPLR+MAP method showed that mel-cepstral distance and RMS logF0 error converged to values comparable to ML estimation when using the same tying topology, demonstrating the effectiveness of the MAP refinement.
- Impact of Training Data Size: As the number of training sentences for the average voice model increased, both objective metrics (mel-cepstral distance and RMS logF0 error) improved. Notably, when the average voice model was trained with more than 1350 sentences, the adapted model outperformed the SD model in RMS logF0 error.
- Subjective Evaluation: Comparison Category Rating (CCR) tests indicated that synthesized speech from the adapted average voice models was rated as more natural than that from SD models when the average voice model was trained with over 1350 sentences.
- Tying Topology Complexity: The number of leaf nodes in the decision tree (representing tying topology complexity) increased with the amount of training data for the average voice model, suggesting that these models can capture more nuanced information.
Limitations and Future Work
- Limitations: The paper acknowledges that the MAP estimation does not perfectly approach ML estimation of the SD model if the tying topologies differ. The performance improvement is contingent on the size and quality of the average voice model training data.
- Future Work: Not explicitly detailed in the provided text, but the findings suggest further exploration into optimizing tying topologies and investigating the impact of different adaptation algorithms in conjunction with MAP estimation. The paper's success with HSMMs also opens avenues for applying this technique to other acoustic modeling frameworks.
Summary updated