Robust word recognition using articulatory trajectories and gestures

Vikramjit Mitra, Hosung Nam, Carol Espy-Wilson, Elliot Saltzman, Louis GoldsteinInterspeech 2010

TL;DR

This paper proposes a novel approach to improve automatic speech recognition (ASR) robustness in noisy environments by incorporating articulatory information (tract variables and gestures) derived from speech acoustics. By training neural networks on synthetic articulatory data and then applying them to natural speech, they demonstrate significant improvements in word recognition accuracy, especially under noisy conditions.

What Problem Does This Paper Address?

Current state-of-the-art ASR systems, primarily based on acoustic features like Mel-frequency cepstral coefficients (MFCCs), struggle with speech variability, particularly coarticulation and noise. Articulatory Phonology (AP) offers a framework where speech is viewed as a set of gestures by vocal tract organs. The hypothesis is that this articulatory information, which is thought to be more invariant than acoustic signals, can help ASR systems overcome noise and variability. However, a significant challenge has been the lack of readily available, natural speech datasets with ground-truth articulatory information, making it difficult to train and evaluate such systems.

Methods

The authors tackle this challenge by first generating a synthetic speech dataset using the TADA (Tract Dynamic Application) model, which links articulatory gestures to acoustic waveforms. This synthetic data, AUR-SYN, is free from measurement noise and inconsistencies.

They then train two neural network-based models:

  1. TV-estimator: A feedforward (FF) Artificial Neural Network (ANN) that estimates tract variables (TVs) – continuous time functions representing vocal tract constriction degree and location – from acoustic features (MFCCs).
  2. Gesture-recognizer: A two-stage cascaded ANN architecture. The first stage uses a non-linear autoregressive ANN (AR-ANN) for gestural activation detection (onset/offset), and the second stage uses an FF-ANN for gestural parameter estimation (target and stiffness).

These "synthetic-data trained" models are then applied to natural speech from the Aurora-2 corpus. Finally, the estimated articulatory information (TVs and recognized gestures) is concatenated with standard acoustic features (MFCCs and RASTA-PLP) and fed into an HMM-based word recognition system to evaluate its noise robustness.

Datasets

  • AUR-SYN: A newly generated synthetic speech dataset. It contains 960 utterances from the Aurora-2 training data, with corresponding ground-truth tract variables (TVs) and gestures generated by the TADA model. This dataset was split into training (70%), development (10%), and testing (20%) sets.
  • Aurora-2 Corpus: A standard dataset for noise-robust speech recognition, consisting of connected digits spoken by American English talkers. The authors used the training and testing sets (specifically test sets A and B, which contain various noise types) for evaluating their ASR system.

Key Insights and Ideas

  • Articulatory Information as a Complement: The study strongly suggests that articulatory information (TVs and gestures) is not a replacement for acoustic features but rather a complementary source of information that significantly enhances ASR performance, especially in noisy conditions.
  • Synthetic Data for Training: The use of a synthetic articulatory database (AUR-SYN) is a crucial enabler, overcoming the lack of natural articulatory data for training inversion models. This approach allows for robust initial training of the TV-estimator and gesture-recognizer.
  • Gesture Recognition Approach Matters: The paper highlights that the specific approach to gesture recognition impacts its effectiveness. Approach-1, which utilizes both acoustic observations (MFCCs) and estimated TVs, yielded the best gesture recognition accuracy, suggesting the benefit of multi-stream information.
  • Tract Variables vs. Gestures: While both TVs and gestures contribute to improved ASR, the results indicate that using them together provides the most significant benefit, reinforcing the idea that they capture different but complementary aspects of speech production.

Key Results

  • TV Estimation Accuracy: The TV-estimator achieved high Pearson product-moment correlation coefficients (PPMC) for estimating tract variables, with values generally above 0.97, indicating good agreement with ground truth.
  • Gesture Recognition Accuracy: The best gesture recognition approach (Approach-1) achieved an overall accuracy of 93.66% on the synthetic data.
  • Word Recognition Improvement: When the estimated articulatory information (TVs and gestures) was combined with MFCCs or RASTA-PLP features, word recognition accuracy in noisy conditions (0-20dB SNR) improved significantly. For example, with MFCCs, accuracy increased from 51.04% to 70.37% (with TVs) and further to 72.47% (with TVs and gestures) at 0dB SNR. Similar improvements were observed for RASTA-PLP.
  • Complementary Information: The results clearly show that neither acoustic features nor articulatory information alone provided the best results. Their combination was essential for achieving superior noise robustness.

Limitations and Future Work

  • Acoustic Mismatch: A significant limitation acknowledged is the potential acoustic mismatch between the clean synthetic training data and the clean/noisy natural speech from different speakers used for testing. This mismatch might limit the accuracy of the TV-estimator and gesture-recognizer on natural speech.
  • Limited Training Data: The AUR-SYN dataset, while useful, represents only about 11% of the Aurora-2 training corpus. Training with a larger, natural speech dataset containing ground-truth articulatory information would likely yield even better results.
  • Future Work: The authors plan to develop a natural speech database with TV and gestural information, which would enable them to build more robust TV-estimators and gesture-recognizers for real-world applications. They are also exploring the use of different machine learning strategies for TV estimation.

Summary updated

BibTeX
@inproceedings{mitra2010robust,
  author = {Vikramjit Mitra and Hosung Nam and Carol Espy-Wilson and Elliot Saltzman and Louis Goldstein},
  title = {{Robust word recognition using articulatory trajectories and gestures}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure