Multi-Corpus Acoustic-to-Articulatory Speech Inversion

Nadee Seneviratne, Ganesh Sivaraman, Carol Espy-WilsonInterspeech 2019

TL;DR

This paper tackles the challenge of creating more generalized speech inversion systems by training a multi-task deep learning model on data from three distinct articulatory speech datasets. By combining diverse data sources, they demonstrate improved cross-dataset generalization, a crucial step towards robust real-world applications of speech inversion.

What Problem Does This Paper Address?

Speech inversion (SI) is the process of inferring how a person's vocal tract is moving (articulatory dynamics) from the sound of their speech. This is a complex problem because the relationship between acoustics and articulation is non-linear and not one-to-one. Traditionally, SI systems have been trained on single datasets, which limits their ability to generalize to new speakers or different recording conditions. This is largely due to variations in speaker anatomy and sensor placement even when using the same measurement technology (like EMA or X-ray microbeam). To build more robust and widely applicable SI systems, researchers need to overcome this data dependency.

Methods

The authors propose a multi-task learning approach using a deep neural network (DNN). The core idea is to train a single model that can predict articulatory movements for multiple datasets simultaneously.

  • Acoustic Feature Extraction: Mel-Frequency Cepstral Coefficients (MFCCs) are extracted from the audio signal using a standard Hamming window and frame shift. To capture temporal context, these MFCCs are contextualized by concatenating frames from a 340ms window.
  • Multi-Task DNN Architecture: A feed-forward DNN with shared hidden layers is employed. Each "task" in this multi-task learning setup corresponds to predicting the articulatory variables (Tract Variables or TVs) for one of the three datasets. This shared architecture allows the model to learn common articulatory patterns across different data sources.
  • Kalman Smoothing: To mitigate high-frequency noise in the estimated articulatory trajectories, a Kalman filter is applied as a post-processing step. This acts as a low-pass filter, leveraging the inherent kinematic smoothness of speech production.

Datasets

The paper utilizes three distinct articulatory speech datasets:

  • X-Ray Microbeam (XRMB) dataset: Collected from 56 speakers (male and female) using X-ray microbeam cinematography. Articulatory movements are tracked via pellets on lips, tongue, and jaw. Tract Variables (TVs) are derived from these trajectories.
  • EMA-IEEE dataset: Recorded from 8 speakers (4 male, 4 female) using a 5-D electromagnetic articulometry (EMA) system. This dataset contains 720 phonetically balanced sentences recorded at both normal and fast rates. 9 Tract Variables (TVs) are computed.
  • MOCHA-TIMIT dataset: Features simultaneous speech and EMA data from one male and one female speaker of British English. Similar to the EMA-IEEE dataset, 9 TVs are derived.

Key Insights and Ideas

  • The Power of Data Diversity: The central insight is that combining data from different sources, even with different measurement techniques, significantly improves the generalization capabilities of speech inversion systems.
  • Multi-Task Learning for Generalization: By forcing a single DNN to learn from multiple datasets, the model is encouraged to find more robust and generalizable representations of articulatory movements, rather than overfitting to the idiosyncrasies of a single dataset.
  • Cross-Dataset Performance Boost: The multi-task model not only performs well on its "native" datasets but also shows improved performance when tested on data from other datasets, indicating better cross-domain adaptation.
  • Specific TV Improvement: The paper highlights that certain articulatory variables (like TBCL and TBCD) see significant improvements in correlation when using the multi-corpus approach, suggesting that some articulatory features are better learned through diverse data.

Key Results

The paper demonstrates the effectiveness of the multi-task learning approach through cross-correlation evaluations:

  • Single-Corpus Baseline: The best single-corpus models achieved average validation correlations ranging from 0.730 (MOCHA-TIMIT) to 0.826 (EMA-IEEE).
  • Multi-Corpus Performance: The multi-task model achieved average validation correlations of 0.768 (XRMB), 0.819 (EMA-IEEE), and 0.780 (MOCHA-TIMIT).
  • Cross-Dataset Generalization:
    • When testing the XRMB single-corpus model on EMA-IEEE data, the correlation dropped significantly (e.g., 0.453 for XRMB test set on EMA-IEEE model).
  • The multi-corpus model showed substantial improvements in cross-dataset testing. For instance, when testing the XRMB test set on the EMA-IEEE output of the multi-corpus model, the correlation increased by 6.9% compared to the single-corpus EMA-IEEE model. Similarly, testing on the MOCHA-TIMIT output saw a 29.5% increase.
  • Similar improvements were observed when testing EMA-IEEE and MOCHA-TIMIT datasets on different outputs of the multi-corpus model.

Limitations and Future Work

  • Limitations: The authors acknowledge that while the multi-corpus model improves generalization, there might be a slight reduction in correlation for the "native" output of a specific dataset compared to a model trained solely on that dataset. This is attributed to the model learning a more generalized representation that might not capture every nuance of a single dataset.
  • Future Work: The authors plan to explore different neural network architectures and alternative acoustic feature representations to further enhance the performance of the speech inversion system.

Summary updated

BibTeX
@inproceedings{seneviratne2019multi,
  author = {Nadee Seneviratne and Ganesh Sivaraman and Carol Espy-Wilson},
  title = {{Multi-Corpus Acoustic-to-Articulatory Speech Inversion}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure