Deep architectures for articulatory inversion

Benigno Uria, Iain Murray, Steve Renals, Korin RichmondInterspeech 2012

TL;DR

This paper explores using deep neural networks for articulatory inversion, the challenging task of predicting vocal tract articulator positions from speech audio. They demonstrate that deep architectures significantly outperform shallow ones, achieving a new state-of-the-art in this domain. This suggests deep learning is a powerful tool for understanding and modeling speech production.

What Problem Does This Paper Address?

The core problem is articulatory inversion: inferring the precise physical movements of the vocal tract (like tongue, lips, jaw) from the sound of speech. This is a notoriously difficult, non-linear, and ill-posed problem because many different articulator configurations can produce the same acoustic signal. Understanding this relationship is crucial for advancing fields like speech recognition (improving accuracy by incorporating production knowledge), speech synthesis (creating more natural-sounding voices), and character animation (automating facial movements for virtual characters). While traditional machine learning methods have been applied, this paper investigates whether the recent successes of deep architectures in other speech tasks can translate to this complex regression problem.

Methods

The authors implement and compare two deep architectures for articulatory inversion:

  • Deep Neural Network (DNN): This is a standard feedforward neural network with multiple hidden layers.
  • Deep Trajectory Mixture Density Network (TMDN): This builds upon the DNN by using a mixture of Gaussian distributions to model the output. This allows the network to represent uncertainty in the articulator positions, which is critical for an ill-posed problem.

A key methodological innovation is the use of unsupervised pretraining with Restricted Boltzmann Machines (RBMs). This technique initializes the weights of the deep network layers sequentially, starting with a Gaussian-Bernoulli RBM for the first layer (handling real-valued acoustic inputs) and then stacking further RBMs. This pretraining helps overcome the challenges of training very deep networks, preventing them from getting stuck in poor local optima and leading to better initial configurations. After pretraining, the entire network is fine-tuned using stochastic gradient descent to minimize the mean squared error (for DNNs) or the negative log-likelihood (for TMDNs). For the TMDN, a post-processing step using the Maximum Likelihood Parameter Generation (MLPG) algorithm is used to infer smooth trajectories from the predicted probability distributions.

Datasets

The primary dataset used is the MNGUO dataset. This dataset consists of parallel recordings of acoustic speech data and electromagnetic midsagittal articulography (EMA) data, which precisely tracks the positions of vocal tract articulators. The MNGUO dataset is noted for its high accuracy compared to other EMA datasets. The paper uses a standard split for training, validation, and testing.

Key Insights and Ideas

  • Depth Matters: The most significant finding is that deep architectures (with multiple hidden layers) consistently outperform shallow, single-hidden-layer networks for articulatory inversion. This suggests that the increased expressive power of deep models, rather than just a larger number of parameters, is responsible for the improved accuracy.
  • Pretraining is Crucial: Unsupervised pretraining with RBMs is essential for achieving good performance with deep networks. Networks initialized randomly performed significantly worse than those that underwent pretraining.
  • TMDNs Offer Superior Uncertainty Modeling: The Deep Trajectory Mixture Density Network (TMDN) achieved better inversion accuracies than the standard Deep Neural Network, even when the DNN's outputs were smoothed. This highlights the benefit of explicitly modeling output uncertainty in ill-posed regression tasks.
  • Deep Architectures Define New State-of-the-Art: The paper establishes new benchmark results for articulatory inversion, demonstrating the effectiveness of deep learning for this complex, time-varying regression problem.

Key Results

  • The best Deep Trajectory Mixture Density Network (TMDN) model achieved an average Root Mean Squared Error (RMSE) of 0.885 mm on the MNGUO test dataset.
  • A Deep Neural Network (DNN) with 5 hidden layers achieved an average RMSE of 0.942 mm.
  • These results represent significant reductions in RMSE compared to previous state-of-the-art methods (e.g., Richmond's shallow TMDN with an RMSE of 0.99 mm).
  • The paper also shows that networks with more hidden layers generally performed better, with the optimal number of layers being around 5-6 for the tested configurations.
  • Randomly initialized networks performed worse, with an average RMSE increase of 0.051 mm for DNNs and 0.054 mm for TMDNs.

Limitations and Future Work

  • Gaussian Mixture Components: The TMDN experiments primarily used a single Gaussian component per mixture. The authors note that Richmond's previous work used a variable number of components per articulator, which could be an area for future improvement.
  • Model Complexity: While deep architectures are beneficial, the optimal number of layers and units per layer still requires careful tuning.
  • Future Directions: The paper suggests that this work is a first demonstration of applying deep architectures to complex, time-varying regression problems like articulatory inversion. Future work could explore more complex mixture models, different pretraining strategies, and potentially end-to-end training without pretraining if computational resources allow.

Summary updated

BibTeX
@inproceedings{uria2012deep,
  author = {Benigno Uria and Iain Murray and Steve Renals and Korin Richmond},
  title = {{Deep architectures for articulatory inversion}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure