TL;DR
This paper presents a novel method for "cross-speaker acoustic-to-articulatory inversion," enabling a system to predict how a reference speaker articulates speech based on the audio of a different speaker. This is crucial for developing more personalized pronunciation training tools that can adapt to individual users.
What Problem Does This Paper Address?
Pronunciation training often benefits from visual feedback showing how the mouth and tongue move to produce sounds. Creating such feedback typically requires mapping speech audio to articulatory movements. While this mapping has been explored, a significant challenge is adapting it to work across different speakers. Ideally, a pronunciation training system should be able to take the speech of any user (the "source speaker") and generate articulatory animations as if a trained, reference speaker were saying it. This allows the system to provide a consistent, high-quality articulatory model for comparison, regardless of the user's voice. Previous methods often struggled with this cross-speaker adaptation, leading to less accurate articulatory predictions.
Methods
The authors propose a "continuous mapping technique" that improves upon their previous "baseline technique."
- Baseline Technique: This method used separate Hidden Markov Models (HMMs) for acoustic and articulatory features, modeling them independently for each phonetic class. The inversion involved two steps: first, decoding the phonetic sequence from the audio, and second, synthesizing articulatory trajectories from the decoded phonemes and HMM states using the MLPG algorithm. Speaker adaptation was done by adjusting acoustic HMM parameters with MLLR. A drawback was that the articulatory synthesis relied heavily on the decoded phonemes, not directly on the acoustic observations.
- Continuous Mapping Technique (Proposed): This new approach models acoustic and articulatory features jointly within each phonetic class using a single-stream "full-covariance" HMM. This allows for a more explicit modeling of the local correlations between acoustic and articulatory features. The inversion process still involves phonetic decoding, but the subsequent synthesis of articulatory trajectories explicitly considers both the decoded phonetic sequence and the acoustic observations. This is achieved by adapting a GMM-based mapping approach to the HMM framework.
- Speaker Adaptation: For cross-speaker scenarios, the authors employ a voice conversion approach based on trajectory GMMs. The goal is to modify the source speaker's acoustic observations so they sound like the reference speaker's, rather than adapting the HMM model parameters directly. This is necessary because adapting full-covariance HMMs would require both audio and articulatory adaptation data, which is impractical.
Datasets
- Reference Speaker Data (PB): Synchronously recorded audio and articulatory data (using Carstens 2D EMA system) from a reference speaker ("PB"). This dataset includes VCVs, CVC words, and sentences, totaling approximately 17 minutes of speech. Articulatory data (EMA coil positions) were downsampled to 100 Hz and low-pass filtered at 20 Hz. Audio was parameterized into 25 mel-cepstrum coefficients.
- Source Speaker Data (TH): Audio-only data from a second speaker ("TH") who pronounced the same text material as the reference speaker. This data was used for cross-speaker experiments and adaptation.
Key Insights and Ideas
- Joint Modeling is Key: Explicitly modeling the local correlations between acoustic and articulatory features within phonetic classes (as done in the continuous mapping technique) leads to more accurate articulatory trajectory estimation, especially when acoustic observations are directly incorporated into the synthesis step.
- Voice Conversion for Adaptation: Using voice conversion to adapt the acoustic observations of the source speaker to match the reference speaker is a viable and practical strategy for cross-speaker adaptation, avoiding the need for articulatory adaptation data.
- Trade-offs in Mapping: Both fine-grained phonetic partitioning (baseline) and less precise partitioning with local regression functions (continuous mapping) can yield similar results for within-speaker inversion, suggesting flexibility in how the articulatory-acoustic space is modeled.
- Importance of Acoustic Input for Synthesis: The baseline technique's drawback of not using acoustic observations during synthesis highlights the need for this direct link to capture acoustic variability and improve the realism of generated articulatory movements.
Key Results
- Within-Speaker Inversion (Reference Speaker PB): Both the baseline and continuous mapping techniques achieved comparable results: 84% audio recognition accuracy, 1.48mm RMS error for articulatory trajectories, and 80.2% articulatory recognition accuracy.
- Cross-Speaker Inversion:
- Phonetic Decoding: MLLR adaptation (for acoustic HMMs) slightly outperformed GMM-based adaptation for phonetic decoding (80% vs. 73% audio accuracy).
- Articulatory Synthesis: The baseline technique (with MLLR adaptation) yielded better articulatory accuracy (77%) than the continuous mapping technique (with GMM-based adaptation, 67%).
- Combined Approach: When MLLR adaptation was used for phonetic decoding and GMM-based voice conversion was used for acoustic adaptation in the continuous mapping technique, the performance became nearly comparable to the baseline technique (73% vs. 77% articulatory accuracy).
Limitations and Future Work
- Limitations:
- For cross-speaker experiments, the accuracy of the estimated articulatory trajectories was lower than for within-speaker inversion.
- The continuous mapping technique, when combined with GMM-based speaker adaptation, showed slightly lower articulatory accuracy compared to the baseline technique with MLLR adaptation.
- No articulatory data was available for the source speaker (TH), necessitating the use of an "articulatory recognition" paradigm for evaluation in cross-speaker scenarios.
- Future Work:
- Investigate the continuous HMM-based mapping technique for both cross-speaker and cross-language acoustic-to-articulatory inversion.
- Test the system's ability to handle speech from foreign speakers and pathological speech.
Summary updated