Abstract
This paper presents a method to produce a new vowel by articulatory control in hidden Markov model (HMM) based parametric speech synthesis. A multiple regression HMM (MRHMM) is adopted to model the distribution of acoustic features, with articulatory features used as external auxiliary variables. The dependency between acoustic and articulatory features is modelled by a group of linear transforms that are either estimated context-dependently or determined by the distribution of articulatory features. Vowel identity is removed from the set of context features used to ensure compatibility between the context-dependent model parameters and the articulatory features of a new vowel. At synthesis time, acoustic features are predicted according to the input articulatory features as well as context information. With an appropriate articulatory feature sequence, a new vowel can be generated even when it does not exist in the training set. Experimental results show this method is effective in creating the English vowel [ʌ] by articulatory control without using any acoustic samples of this vowel.
Introduction
Standard HMM-based parametric speech synthesis systems are inherently data-driven, relying on large corpora of recorded speech to learn the mapping between linguistic labels and acoustic features. A significant challenge arises when the training data is incomplete or when a specific phonetic sound—such as a particular vowel—is missing from the corpus. Traditional systems often fail to "extrapolate" these missing sounds because they rely heavily on explicit vowel identity labels. If the system has never encountered the label for the vowel /ʌ/ (as in "but"), it cannot produce it accurately.
This research explores a novel solution: using phonetic knowledge of articulatory movements to bridge the gap. By modeling how the physical movement of the mouth (tongue, lips, etc.) relates to sound, the researchers aim to synthesize entirely new vowels that were never present in the training data.
Methodology
The proposed system treats acoustic features as dependent variables of physical articulatory movements using a Multiple Regression Hidden Markov Model (MRHMM).
Multiple Regression HMM (MRHMM)
Unlike a standard HMM where the mean of a sound's distribution is fixed for a given state, the MRHMM employs a linear regression model. Here, the acoustic output is calculated based on "external" articulatory features, such as the trajectories of the tongue and lips.
Feature-Space-Switched MRHMM
To enhance flexibility, the researchers introduced a "feature-space-switched" approach (MR-TF). This uses a Gaussian Mixture Model (GMM) to cluster the articulatory space. Instead of the regression being tied strictly to a linguistic "state," the system switches regression matrices based on the physical position of the articulators. This allows the system to handle articulatory configurations that might conflict with the linguistic states learned from other vowels.
Context Feature Tailoring
In standard synthesis, "vowel identity" is a primary feature. To prevent the model from relying on memorized labels, the authors explicitly removed vowel identity from the context features during training. This forces the model to rely solely on the relationship between physical movement and sound.
Experimental Setup
The study utilized a multi-channel articulatory database consisting of synchronized acoustic and Electromagnetic Articulography (EMA) data. The EMA data tracked six sensors placed on the tongue (dorsum, body, tip), lips (upper, lower), and lower incisor in 2D space.
To simulate "vowel creation," the researchers removed all 809 sentences containing the vowel /ʌ/ from the training set. They then attempted to synthesize /ʌ/ using only articulatory descriptions.
Results and Discussion
The performance of the "created" vowel was evaluated against standard HMM baselines using both objective and subjective measures.
Objective Accuracy
The MRHMM systems achieved significantly lower Spectral Root Mean Squared Error (RMSE) for the unseen vowel /ʌ/ compared to standard HMMs. This indicates that the regression-based approach more accurately captures the acoustic properties of the missing sound.
Subjective Perception
In listening tests involving 32 native speakers, the results were striking. The vowel created by the proposed MR-TF system was correctly identified 66.25% of the time. In comparison, the standard HMM baseline (STD-F), which had not seen the vowel during training, was only identified correctly 35% of the time.
Notably, the 66.25% accuracy for the created vowel was nearly equal to the recognition rates for vowels that were included in the training set, such as /ɛ/ (68.75%).
Conclusion
This research demonstrates that articulatory features provide a "universal bridge" for speech synthesis. By decoupling acoustic output from linguistic labels and anchoring it in physical movement, it is possible to generate high-quality, recognizable speech sounds that the system has never "heard" before. Future work will focus on predicting these articulatory trajectories from text alone, potentially enabling zero-shot synthesis for low-resource languages.
Summary updated