Acoustic-to-articulatory inversion based on local regression

Samer Al Moubayed, G. AnanthakrishnanInterspeech 2010

TL;DR

This paper introduces a novel acoustic-to-articulatory inversion method using local regression, outperforming traditional global models like Gaussian Mixture Models (GMMs) in accuracy. By focusing on local relationships between speech acoustics and vocal tract movements, the method achieves more precise estimations of articulation, with potential applications in speech synthesis and recognition.

What Problem Does This Paper Address?

Understanding the relationship between the sounds we produce (acoustics) and the physical movements of our vocal tract (articulation) is a long-standing challenge in speech science. This "acoustic-to-articulatory (A-to-A) inversion" is crucial for applications like low-bitrate speech coding, driving realistic speech avatars, and improving speech recognition systems. Traditional methods often rely on "inversion by synthesis," which is limited by the realism of the articulatory synthesizer used. More recent approaches leverage large datasets of simultaneous acoustic and articulatory recordings to train statistical models. However, many of these methods assume a single, global mapping function between acoustics and articulation, which is known to be highly non-linear and potentially non-unique. This paper addresses the limitations of global models by proposing a "memory-based" or "local regression" approach that builds a model specific to the local neighborhood of a given acoustic input, better capturing the complex, non-linear nature of the acoustic-articulatory mapping.

Methods

The core of this paper is the application of local regression for A-to-A inversion. Instead of learning a single global function, local regression builds a model based on the "neighborhood" of a test sample – the most similar acoustic frames in the training data. Two types of local regression are explored:

  • Non-parametric Regression: This method directly estimates the articulatory parameters by taking a weighted average of the articulatory positions of the K nearest acoustic neighbors. The weights are determined by the inverse distance to the acoustic neighbors, meaning closer neighbors have a stronger influence. This is essentially a form of Kernel Regression.
  • Local Linear Regression: This approach assumes a linear relationship between acoustics and articulation within the local neighborhood. It estimates local linear regression parameters (slope and intercept) from the K nearest neighbors to predict the articulatory parameters.

A key innovation is the incorporation of Maximum Likelihood Trajectory Estimation (MLTE). This technique augments the estimated articulatory positions with dynamic features (like velocity and acceleration) derived from the acoustic signal. This allows the model to capture the temporal continuity and dynamics of articulation, leading to smoother and potentially more accurate trajectories. The MLTE is then found by solving a weighted least-squares problem, where weights are based on the density of neighbors.

The paper also compares these local methods against a Gaussian Mixture Model Regression (GMMR), a representative global model.

Datasets

The experiments were conducted using the MOCHA database. This dataset contains simultaneous acoustic and articulatory data from a single female speaker reading 460 TIMIT sentences. The articulatory data includes X- and Y-axis trajectories of 7 EMA coils placed on various parts of the vocal tract (lower jaw, lips, tongue, velum). The acoustic features used were Mel Frequency Cepstral Coefficients (MFCCs), augmented with temporal context and reduced using Principal Component Analysis (PCA). A ten-fold cross-validation scheme was employed, with 314 sentences for training and 46 for testing in each fold.

Key Insights and Ideas

  • Local is Better (for Accuracy): Local regression methods, particularly local linear regression, demonstrate superior accuracy in estimating articulatory positions compared to global GMMR, especially when incorporating articulatory dynamics.
  • Dynamics Matter: Incorporating articulatory dynamics (velocity, acceleration) via MLTE significantly improves the accuracy of the inversion, particularly for local linear regression. This highlights the importance of temporal information in speech production.
  • Non-parametric vs. Local Linear: While both local methods outperform GMMR, local linear regression with dynamics shows a notable advantage. However, local linear regression requires a much larger neighborhood (K) to achieve optimal results compared to the non-parametric approach.
  • Neighborhood Size is Crucial: The choice of K (number of neighbors) is critical. For non-parametric regression, a small K (around 20) is optimal, while for local linear regression, a much larger K (around 1000) is needed. This difference is attributed to the sensitivity of linear regression to noise, necessitating more data for stable parameter estimation.
  • Trade-off Between Accuracy and Speed: Local regression methods are significantly slower than GMMR (around 100 times slower) but have no training time, making them potentially suitable for offline applications where accuracy is paramount.

Key Results

  • MMSE Estimates (No Dynamics):
    • Non-parametric regression: Average RMSE of 1.56 mm.
  • Local linear regression: Average RMSE of 1.52 mm.
  • Both local methods showed statistically significant improvements in Correlation Coefficient (mCC) over GMMR.
  • MLTE Estimates (With Dynamics):
    • Local linear regression with MLTE achieved an optimum performance of 1.52 mm mRMSE and an mCC of 0.78.
  • Non-parametric regression with MLTE achieved an optimum performance of 1.56 mm mRMSE and an mCC of 0.76.
  • Local linear regression with MLTE showed statistically significant improvements over GMMR for both mRMSE (p<0.1) and mCC (p<0.05).
  • Comparison to Prior Work: While the proposed methods show good performance, they fall within the variation observed for prior work using similar techniques. For instance, Richmond [5] reported 1.4 mm mRMSE with trajectory mixture density networks, and Toda et al. [15] reported 1.45 mm mRMSE with GMMR and dynamic features. The authors note that the training/testing split for these studies is unknown.

Limitations and Future Work

  • Computational Cost: Local regression methods are computationally expensive, particularly the nearest neighbor search, making them around 100 times slower than GMMR.
  • Generalization: The current study was conducted on a single speaker. Future work aims to apply the method to other speakers and databases to assess its generalizability.
  • Relative Importance of Dynamics: Further investigation is needed to quantify the exact importance of articulatory dynamics versus acoustic neighborhood information.
  • Speed Optimization: Developing algorithms to speed up the nearest neighbor search (e.g., using pruning, convex clustering, or hierarchical trees) is a key direction for future work.
  • Speech Recognition Applications: Exploring whether this A-to-A inversion method can improve speech recognition by providing articulatory knowledge is another promising avenue.

Summary updated

BibTeX
@inproceedings{moubayed2010acoustic,
  author = {Samer Al Moubayed and G. Ananthakrishnan},
  title = {{Acoustic-to-articulatory inversion based on local regression}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure