Acoustic-to-Articulatory Mapping Based on Mixture of Probabilistic Canonical Correlation Analysis

Hidetsugu Uchida, Daisuke Saito, Nobuaki MinematsuInterspeech 2017

TL;DR

This paper introduces a novel acoustic-to-articulatory (a2a) mapping model called Mixture of Probabilistic Canonical Correlation Analysis (mPCCA). By leveraging a structured covariance matrix approach, mPCCA aims to improve the accuracy of predicting vocal tract movements from speech sounds, outperforming traditional Gaussian Mixture Model (GMM) methods, especially when dealing with complex data structures.

What Problem Does This Paper Address?

Estimating articulatory movements (how the vocal tract shapes) from acoustic speech signals is a challenging but important problem in speech processing. This "acoustic-to-articulatory" (a2a) mapping has applications in areas like speech synthesis, understanding speech disorders, and even silent speech recognition. Existing methods, such as those based on Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs), model the relationship between acoustic features and articulatory movements using linear transformations. However, these methods can suffer from overfitting, especially when the number of model components increases to capture more complex data patterns. While some advanced models like Matrix Variate Gaussian Mixture Models (MVGMMs) address overfitting by structuring covariance matrices, they are limited to mapping data within the same domain (e.g., acoustic-to-acoustic), making them unsuitable for a2a mapping. This paper bridges this gap by proposing a novel model that combines the benefits of structured covariance matrices for avoiding overfitting with the flexibility to map between different domains like acoustics and articulation.

Methods

The core of the proposed method is the Mixture of Probabilistic Canonical Correlation Analysis (mPCCA). mPCCA builds upon Probabilistic Canonical Correlation Analysis (PCCA), which assumes that observed data in different domains (acoustic and articulatory) are generated from a common underlying latent variable. This latent variable is hypothesized to capture shared factors inherent in both domains.

mPCCA extends PCCA by using a mixture model, allowing it to capture more complex relationships. Crucially, mPCCA structuralizes the covariance matrices of the joint probability distribution of acoustic and articulatory data. This structuralization is achieved through the transformation coefficients of the linear mappings from the latent variable to the observed data. This approach is designed to prevent overfitting, even with an increased number of mixture components. The training and mapping processes are derived using the Expectation-Maximization (EM) algorithm.

The paper also proposes two mapping processes:

  1. mPCCA-w/:Init: This process requires an initial estimate of the articulatory feature.
  2. mPCCA-w/o:Init: This process is an approximation that does not require an initial estimate, making it more practical.

Datasets

The experiments were conducted using the MOCHA-TIMIT database. This database contains parallel acoustic and articulatory data from one male and one female speaker. The articulatory data includes measurements from seven points on the vocal tract (e.g., tongue tip, lips, velum), each represented by 2D coordinates. Acoustic features were extracted as 24-dimensional mel-cepstrums, which were then reduced to 75 principal components. The data was split into 5 parts for 5-fold cross-validation, with 4 parts for training and 1 part for testing.

Key Insights and Ideas

  • Bridging the Domain Gap: The primary innovation is adapting a method (PCCA) that inherently handles relationships between different data domains for the specific task of acoustic-to-articulatory mapping, overcoming limitations of previous domain-specific models.
  • Structured Covariance for Overfitting Prevention: By imposing a structure on covariance matrices, mPCCA effectively reduces the number of free parameters, making the model more robust to overfitting compared to standard GMMs, especially as model complexity (number of components) increases.
  • Latent Variable Interpretation: The latent variable in mPCCA is hypothesized to represent a common underlying factor that influences both speech acoustics and vocal tract articulation, offering a potentially more interpretable model of speech production.
  • Practical Mapping without Initialization: The development of the mPCCA-w/o:Init mapping process is a significant practical advancement, as it removes the dependency on an initial articulatory estimate, simplifying its application.

Key Results

The proposed mPCCA-based mapping model demonstrated improved performance over conventional GMM-based mapping in terms of estimation error (measured by Root Mean Square Error - RMSE). Specifically:

  • mPCCA-w/:Init consistently achieved lower RMSE than GMM-mmse (GMM with minimum mean square error criterion) across both speakers, particularly as the number of components increased.
  • Table 1 shows that with 64 components, mPCCA-w/:Init achieved an RMSE of 1.56 mm for the female speaker and 1.48 mm for the male speaker, outperforming GMM-mmse (1.58 mm and 1.51 mm, respectively).
  • The mPCCA-w/o:Init model, while generally inferior to the initialized version and GMM-mmse, still shows promise, suggesting that further refinement of this initialization-free approach could be beneficial. The results indicate that the structuralized covariance matrices effectively mitigate overfitting.

Limitations and Future Work

  • Initialization Dependency: The mPCCA-w/:Init method requires an initial estimate of the articulatory feature, which might not always be readily available or accurate. While mPCCA-w/o:Init addresses this, it showed inferior performance in the experiments.
  • Performance of Initialization-Free Model: The mPCCA-w/o:Init model's performance was noted as being inferior to both GMM-mmse and the initialized mPCCA. This suggests that further research is needed to improve the accuracy of the initialization-free mapping process.
  • Future Work: The authors implicitly suggest that improving the mPCCA-w/o:Init method is a key direction for future work, aiming to achieve comparable or better performance without requiring an initial articulatory estimate. Further exploration into the interpretability of the latent variable could also be a valuable avenue.

Summary updated

BibTeX
@inproceedings{uchida2017acoustic,
  author = {Hidetsugu Uchida and Daisuke Saito and Nobuaki Minematsu},
  title = {{Acoustic-to-Articulatory Mapping Based on Mixture of Probabilistic Canonical Correlation Analysis}},
  booktitle = {Interspeech},
  year = {2017},
}
Figure