TL;DR
This paper introduces a novel "continuous HMM-based mapping" technique for silent speech interfaces. By jointly modeling articulatory (ultrasound/video) and acoustic (speech) features within a single framework, it significantly improves the accuracy of synthesizing audible speech from silent articulatory movements, outperforming previous methods that treated these features separately.
What Problem Does This Paper Address?
The paper tackles the challenge of creating a "silent speech interface" (SSI). These interfaces aim to enable communication for individuals who cannot vocalize speech, such as those who have undergone laryngectomy, or for applications requiring discreet communication in noisy or confidential environments. The core problem is accurately converting silent articulatory movements (captured by sensors like ultrasound and cameras) into intelligible, audible speech. Previous approaches often struggled because they either relied solely on decoded phonetic sequences or treated articulatory and acoustic features independently, leading to a loss of information and degraded speech quality.
Methods
The authors propose a "continuous HMM-based mapping technique" that refines existing Hidden Markov Model (HMM) approaches. The key innovation lies in jointly modeling articulatory and spectral features for each phonetic class using a "full-covariance" HMM. This means that instead of separate models for articulatory and acoustic data, a single model captures their interdependencies.
The mapping process occurs in two steps:
- Phonetic Decoding: Similar to previous methods, an HMM is used to determine the most likely sequence of phonetic states from the observed articulatory data.
- Spectral Trajectory Estimation: This is where the novelty lies. Instead of estimating spectral features solely from the decoded phonetic sequence, the new method explicitly incorporates the articulatory observations into the estimation. This is achieved by deriving a conditional probability distribution that links the spectral features to the articulatory features within each HMM state. The spectral trajectories are then synthesized using an adaptation of the Maximum Likelihood Parameter Generation (MLPG) algorithm, which now considers both the decoded states and the articulatory data.
This contrasts with their "baseline technique," which performed phonetic decoding and then synthesized speech parameters independently of the articulatory observations.
Datasets
The study utilizes a custom-built continuous speech database consisting of one hour of high-speed ultrasound and video sequences, recorded synchronously with audio. This database was collected from a single, native English speaker pronouncing 1132 sentences from the CMU ARCTIC corpus. The data was acquired using the Ultraspeech system. No new benchmarks were introduced; rather, the existing database was used to evaluate the proposed method against a prior approach.
Key Insights and Ideas
- Joint Modeling is Crucial: The most significant insight is that explicitly modeling the local correlations between articulatory movements and acoustic features within a unified framework leads to substantially better speech synthesis. Treating these as separate problems misses vital information.
- Full-Covariance HMMs Unlock Dependencies: The use of full-covariance Gaussian distributions within the HMM states is key to capturing these complex interdependencies, allowing the model to learn how specific articulatory configurations map to specific acoustic characteristics.
- Robustness to Decoding Errors: The continuous mapping technique shows improved robustness to errors in the phonetic decoding stage. This is because even if the phonetic sequence is slightly off, the direct influence of the articulatory data helps to correct or guide the spectral trajectory estimation, mitigating the impact of decoding mistakes.
Key Results
- Objective Evaluation: The continuous HMM-based mapping technique achieved an average improvement of 0.33 dB for Mel-cepstral distance (MCD₁) and 0.51 dB for MCD₀ compared to the baseline method. These improvements were statistically significant.
- Perceptual Evaluation: A listening test (XAB) with 15 sentences revealed that listeners preferred stimuli synthesized with the continuous mapping technique in 80% of cases, indicating a significant perceptual improvement.
- Decoding Accuracy: The paper also reports decoding accuracies of 68.4% for an unconstrained scenario and 78.3% for a constrained scenario (using a dictionary). The continuous mapping technique showed slightly less performance degradation (3.9%) when moving from perfect phonetic alignment to the unconstrained decoding scenario, compared to the baseline (4.2%).
Limitations and Future Work
- Real-time Implementation: The authors acknowledge that the current implementation is not real-time. Future work will focus on achieving real-time performance, potentially by adapting low-delay feature mapping techniques.
- Context-Dependency: Due to data limitations, context-dependent full-covariance HMMs could not be trained directly for the continuous mapping technique. Instead, context-independent HMMs were used for the synthesis stage, relying on context-dependent HMMs from the baseline for phonetic decoding. Future work might explore methods to incorporate more context.
- Data Sparsity: Training full-covariance HMMs is data-intensive. While techniques like state-tying were used for the baseline, the authors note the challenges of training complex models on limited data.
Summary updated