TL;DR
This paper introduces a novel method to improve speech synthesis by learning which parts of the vocal tract (articulators) are most important for producing specific sounds. By weighting these articulators based on their predictive power for each sound, they significantly reduce errors in estimating the acoustic properties of speech.
What Problem Does This Paper Address?
Generating realistic synthetic speech often involves mapping the physical movements of the vocal tract (articulators) to the resulting sound (acoustic spectra). Traditional methods either rely on complex physiological models or statistical approaches that assume all articulators are equally important. However, this assumption is flawed; for example, lip position is crucial for sounds like "p" but less so for sounds made further back in the mouth. This paper tackles the challenge of creating a more accurate and efficient mapping by recognizing that the importance of different articulators varies depending on the specific speech sound (phone) being produced. This is particularly relevant when using limited articulatory data, such as from Electromagnetic Articulography (EMA), where not all vocal tract movements can be precisely tracked.
Methods
The core contribution is a phone-dependent feature weighting scheme. The authors build upon a baseline "codebook" method that finds similar articulatory configurations in a database to predict acoustic features. Their innovation lies in introducing a weighted distance measure for comparing articulatory configurations. Instead of treating all articulators equally, they assign a specific weight to each articulator for each phone.
This weighting is learned using a non-negative least squares (NNLS) approach. For each phone, they calculate the cepstral distance (a measure of acoustic similarity) between frames and then determine how well individual articulatory feature distances predict this cepstral distance. Features that consistently show small distances when the acoustic distance is small, and large distances when the acoustic distance is large, are given higher weights.
To prevent overfitting and ensure a smooth transition from their new method to the baseline, they also introduce a regularization term that allows for a controlled blend between the learned weights and uniform weights.
Datasets
The experiments were conducted using a corpus of Electromagnetic Articulography (EMA) recordings collected at the Centre for Speech Technology Research, University of Edinburgh. This dataset contains synchronized articulatory and acoustic data. The articulatory data was captured using a Carstens AG500 3D-articulograph with six sensors tracking lip and tongue movements. Acoustic data was recorded simultaneously. The corpus was partitioned into sets for weight training, candidate selection, and testing.
Key Insights and Ideas
- Articulators are not created equal: The importance of specific vocal tract movements varies significantly depending on the speech sound. This is a fundamental linguistic concept ("critical articulators") that the authors leverage for acoustic prediction.
- Phone-specific optimization pays off: Learning separate weightings for each phone leads to substantial improvements in acoustic prediction accuracy, outperforming a uniform weighting scheme.
- Lip and tongue tip are often key: The learned weights often highlight the lower lip, tongue tip, and tongue dorsum as particularly important articulators, aligning with linguistic intuition for many sounds.
- Trade-off between acoustic and articulatory accuracy: While the weighted method significantly improves acoustic prediction, it can lead to a slight increase in articulatory prediction error. The authors suggest this is acceptable, as the articulatory features showing increased error are deemed less critical for the overall acoustic output.
Key Results
The proposed phone-dependent weighting scheme reduced overall cepstral error by 6% compared to the baseline method with uniform weights. This improvement was statistically significant. The best performance was achieved with a regularization parameter of $\alpha = 0.8$. Vowels showed the greatest benefit, with improvements observed for 80% of the tested phones. The method also demonstrated that features with high weights (e.g., lower lip for labiodental/bilabial phones) are consistent with known critical articulators.
Limitations and Future Work
The authors acknowledge that their method is based on a linear weighting scheme, which might not fully capture complex, non-linear relationships between articulatory movements and acoustic spectra for certain phones. They also note that the articulatory prediction error increases with their method, suggesting that the features with increased error are less critical for acoustic output.
Future work could involve:
- Incorporating acoustically derived features like pitch and loudness to further complement articulatory information.
- Exploring the use of dynamic articulatory features (velocity and acceleration) to improve the modeling of transient sounds like stops.
- Transforming EMA data into derived measures that capture known linguistic relevance and anatomical constraints, potentially moving towards a more gestural representation of speech production.
- Perceptual evaluation of synthesized utterances to assess the subjective quality of the speech produced by their method.
Summary updated