Predicting unseen articulations from multi-speaker articulatory models

G. Ananthakrishnan, Pierre Badin, Julián Andrés Valdés Vargas, Olov EngwallInterspeech 2010

TL;DR

This paper tackles the challenge of predicting how a person articulates sounds they haven't explicitly produced during data collection, using models trained on multiple speakers. They demonstrate that by analyzing MRI data of vocal tract shapes, they can build models that generalize to unseen articulations, achieving an average prediction error of 5.25 mm for tongue contours. This work is significant for advancing personalized speech synthesis and understanding the fundamental differences in how individuals produce speech.

What Problem Does This Paper Address?

Speech production is inherently variable across individuals due to differences in anatomy (size and shape of articulators) and potentially in the strategies they employ to produce the same sounds. This "inter-speaker variability" is a major hurdle for creating accurate articulatory models that can be used for tasks like speech synthesis or understanding speech production mechanisms. While previous research has explored modeling this variability, this paper specifically addresses the challenge of predicting articulations for sounds a speaker hasn't produced in the training data. This is crucial for building truly personalized and robust articulatory models that don't require exhaustive data collection for every possible sound.

Methods

The authors employ three-mode factor analysis techniques to model the variations in vocal tract shapes across different articulations (phonemes), articulators (tongue, jaw, etc.), and speakers. They compare three main methods:

  • PARAFAC (Parallel Factor Analysis): A standard three-way decomposition method that assumes independence between the modes.
  • Tucker Decomposition (Tucker3): Another three-way decomposition method that allows for more flexibility by introducing a "core matrix" to model interactions between factors.
  • Two-Level PCA: A more conventional approach where the three-dimensional data is first unfolded into a two-dimensional matrix for PCA, followed by a second-level decomposition to capture speaker-specific variations.

A key methodological contribution is the development of an "inversion" technique. Once a multi-speaker model is built, this method allows for the prediction of control parameters (representing articulation) for a new, unseen articulation by essentially reversing the modeling process. They also explore using the Expectation-Maximization (EM) algorithm to handle missing data, particularly for the sub-lingual cavity in tongue contours.

Datasets

The study utilizes a dataset of midsagittal MRI images from three French speakers (two male, one female). These speakers sustained articulations of 73 different phonemes for 16 seconds each. The corpus includes:

  • 10 French oral vowels.
  • 3 nasal vowels.
  • 10 consonants articulated in VCV contexts with six vowels.

The data was processed to extract:

  • Tongue contours: Defined by 200 equidistant points along the tongue from the base of the epiglottis to the jaw attachment.
  • 2D midsagittal vocal tract distance function: An automated method to measure distances between upper and lower vocal tract contours, sampled at 50 points.

Key Insights and Ideas

  • Generalization is Achievable: The core finding is that multi-speaker articulatory models, even when trained on a subset of a speaker's articulations, can effectively predict unseen articulations. This validates the concept of learning general articulatory strategies and anatomical constraints.
  • Optimal Number of Factors: The study provides a methodology for determining the optimal number of factors for these models, suggesting that for their data, three factors are optimal for generalization. This is a significant contribution, as previous studies often chose factors arbitrarily.
  • Modeling Hidden Cavities: The proposed methods, particularly with the EM algorithm, can successfully model and predict parts of the vocal tract that are not visible in the original MRI data, such as the sub-lingual cavity. This leads to more complete articulatory reconstructions.
  • Automated 2D Vocal Tract Function: The development of an automated method for extracting the 2D vocal tract function simplifies data processing and makes it more consistent across speakers and articulations.

Key Results

  • Prediction Accuracy: The average prediction Root Mean Square Error (RMSE) for tongue contours was 5.25 mm, and for the 2D midsagittal vocal tract distances, it was 3.3 mm.
  • Model Comparison: While several three-way analysis methods were compared, the first-level inversion of the unfolded PCA showed the best results for the 2D vocal tract function, and TUCKER3 modeling performed best for midsagittal tongue contours in terms of generalization. However, the authors note that the differences in performance between the methods were not statistically significant.
  • Factor Impact: The number of factors significantly impacts performance, with three factors being identified as optimal for generalization.
  • Qualitative Improvement: Figure 6 demonstrates that the models can predict the shape of the tongue, including previously hidden sub-lingual cavities, even when the original data for that articulation was missing.

Limitations and Future Work

  • Statistical Significance: The authors note that while some methods performed slightly better, the differences were not statistically significant, suggesting that current methods are broadly comparable in their generalization capabilities.
  • Minimum Data Requirement: Future work should focus on determining the minimum amount of articulation data required for a speaker to build a model that generalizes well to all other articulations.
  • Speaker vs. Strategy Variability: Further research is needed to disentangle variability arising from anatomical differences versus variability due to different articulatory strategies.
  • Discriminability: An important aspect to investigate is whether the generalization process affects the ability to distinguish between different phonemes in the articulatory space.

Summary updated

BibTeX
@inproceedings{ananthakrishnan2010predicting,
  author = {G. Ananthakrishnan and Pierre Badin and Julián Andrés Valdés Vargas and Olov Engwall},
  title = {{Predicting unseen articulations from multi-speaker articulatory models}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure