TL;DR
This paper introduces a novel "state mapping" method for cross-lingual speaker adaptation in HMM-based speech synthesis. By mapping acoustic states between languages, their approach significantly improves synthesized speech quality and speaker similarity compared to previous phone-mapping methods, offering a more robust way to synthesize speech in one language with the voice of a speaker from another.
What Problem Does This Paper Address?
The goal of cross-lingual speaker adaptation in speech synthesis is to make synthesized speech in a target language sound like a specific speaker, even if that speaker doesn't speak the target language. Imagine wanting to synthesize English speech that sounds like a particular Japanese speaker. Traditional methods often struggle because they rely on mapping phonetic units (phones) between languages, which can be inaccurate due to phonetic and prosodic differences. This inaccuracy limits how well the speaker's characteristics can be transferred, potentially introducing unwanted language identity from the adaptation data or failing to capture the target speaker's nuances. This paper aims to overcome these limitations by proposing a more granular mapping approach at the acoustic state level.
Methods
The core innovation is the state mapping based method. Instead of mapping phones, this method establishes a correspondence between the acoustic states of HMMs trained for the source (English) and target (Japanese) languages.
Here's how it works:
- Average Voice Models: They first train "Average Voice" models for both the source and target languages. These models represent the general acoustic characteristics of each language.
- KLD-based State Mapping: The crucial step is establishing a mapping between the states of these two models. They use the Kullback-Leibler Divergence (KLD) to measure the similarity between acoustic states. For each state in the target language model, they find the most similar state in the source language model. This creates a "state map."
- Two Adaptation Approaches: Based on this state map, they propose two ways to adapt the source model:
- Data Mapping: This approach uses the state map to "attach" the target language adaptation data to the source language model. Essentially, adaptation data from Japanese is treated as if it were English data, guided by the state map. This is an extension of previous phone-mapping methods but operates at the state level.
- Transform Mapping: This method first trains adaptation transforms on the target language model using Japanese data. Then, the state map is used to transfer these learned transforms from the target language states to the corresponding source language states. This approach aims to avoid directly exposing the source model to the target language's acoustic characteristics during adaptation.
Datasets
- Source Language (English): CMU-ARCTIC English database (approx. 1 hour per speaker for 2 males and 2 females).
- Target Language (Japanese) Average Voice Model: NIT Japanese database (approx. 1 hour per speaker for 2 males and 2 females).
- Target Language Adaptation Data: Japanese speech data from one male speaker (50 utterances).
The paper notes that no English speech data from the target Japanese speaker was available, so a speaker-dependent (SD) model was trained using Japanese data for comparison in the DMOS test.
Key Insights and Ideas
- State-level mapping is superior to phone-level: Mapping at the acoustic state level provides a more fine-grained and accurate correspondence between languages than mapping at the broader phonetic unit (phone) level. This is because states capture more detailed acoustic information.
- Data Mapping vs. Transform Mapping Trade-offs:
- Data Mapping: Achieves better speaker similarity because it directly uses the target speaker's data for adaptation. However, it risks introducing some of the target language's identity into the synthesized speech.
- Transform Mapping: Achieves better speech quality by avoiding direct exposure to target language data. However, it might suffer from a mismatch between the source and target language voice models if they weren't trained under similar conditions, potentially reducing speaker similarity.
- The importance of Average Voice Models: Adapting an "Average Voice" model is generally better than adapting a speaker-dependent model for cross-lingual tasks.
Key Results
The experiments compared three approaches:
- DM-P: Phone mapping based data mapping.
- DM-S: State mapping based data mapping.
- TM-S: State mapping based transform mapping.
The results, based on Mean Opinion Scores (MOS) for speech quality and Direct Mean Opinion Scores (DMOS) for speaker similarity, showed:
- Overall Improvement: Both state mapping approaches (DM-S and TM-S) significantly outperformed the phone mapping approach (DM-P) in terms of synthesized speech quality.
- Data Mapping Advantage: DM-S achieved better speaker similarity.
- Transform Mapping Advantage: TM-S achieved better speech quality.
These findings align with the discussed advantages and disadvantages of each method.
Limitations and Future Work
- Data Mapping's Language Identity Issue: The authors acknowledge that the data mapping approach might still introduce some of the target language's identity into the synthesized speech.
- Transform Mapping's Model Mismatch Issue: The transform mapping approach can suffer from a mismatch between source and target language voice models if they are not trained under similar conditions, potentially affecting speaker similarity.
- Future Work: The authors suggest applying "model space normalization" for the transform mapping approach to mitigate the mismatch issue.
Summary updated