Acoustic and Articulatory Feature Based Speech Rate Estimation Using a Convolutional Dense Neural Network

Renuka Mannem, Jhansi Mallela, Aravind Illa, Prasanta Kumar GhoshInterspeech 2019

TL;DR

This paper introduces a novel Convolutional Dense Neural Network (CDNN) approach for estimating speech rate by combining acoustic and articulatory features. Crucially, it demonstrates that articulatory features, even when estimated from acoustics, significantly improve speech rate estimation accuracy, especially in noisy conditions, outperforming traditional methods.

What Problem Does This Paper Address?

Estimating speech rate – the number of speech units (like syllables) per second – is vital for various speech processing tasks, including speech recognition robustness, second language learner fluency assessment, and even speaker recognition. Existing methods often rely on acoustic features alone or require complex systems (like ASR) to identify syllable boundaries, which can be unreliable in noisy environments or when transcriptions aren't available. Furthermore, articulatory features, which directly reflect the physical movements of the vocal tract, are known to be informative but are difficult to obtain in real-time. This paper tackles the challenge of robust speech rate estimation, particularly in noisy conditions, by exploring the synergistic use of acoustic and articulatory features, and importantly, by investigating how to leverage articulatory information when it's not directly measured.

Methods

The core of this work is a Convolutional Dense Neural Network (CDNN) architecture designed for speech rate estimation. The CDNN processes features extracted from speech. The paper explores three main feature sets:

  • Acoustic Features: Mel Frequency Cepstral Coefficients (MFCCs) are extracted using standard methods.
  • Articulatory Features (Direct): Articulograms, representing the time-varying vocal tract profile, are derived from real-time Magnetic Resonance Imaging (rtMRI) videos. These are computed by measuring distances between points on the upper and lower vocal tract boundaries.
  • Articulatory Features (Estimated): Recognizing that direct articulogram measurement is impractical, the authors employ an Acoustic-to-Articulatory Inversion (AAI) method. This uses a Bidirectional Long Short-Term Memory (BLSTM) network to predict articulogram features from MFCCs.

The CDNN architecture itself consists of convolutional layers to capture spatial and temporal characteristics, followed by dense layers for regression to estimate the syllable rate. The input is segmented into one-second chunks. The network is trained using Mean Squared Error (MSE) loss.

Datasets

The primary dataset used is the USC-TIMIT corpus. This corpus is unique as it provides synchronized audio recordings and rtMRI videos of subjects speaking. The rtMRI videos capture the midsagittal plane of the upper airway. For their experiments, the authors selected data from four subjects (two female, two male), covering 456-460 sentences per subject. The audio is recorded at 20kHz, and rtMRI frames are captured at 23.18 frames/sec.

Key Insights and Ideas

  • Articulatory Features Enhance Robustness: The paper strongly suggests that articulatory features, by capturing the physical dynamics of speech production, offer a more robust signal for speech rate estimation, especially when acoustic signals are noisy.
  • Estimated Articulatory Features are Powerful: The most significant insight is that articulatory features estimated from acoustics (using the AAI method) perform better than using acoustic features alone or even directly measured articulatory features in some cases. This is crucial because it bypasses the need for expensive rtMRI equipment during testing.
  • CDNN Architecture's Efficacy: The CDNN, with its ability to learn complex spatial and temporal patterns, proves effective in integrating these multimodal features for accurate speech rate prediction.
  • Tongue Movement Correlation: The authors observe that estimated articulograms show increased distances in the anterior tongue region during vowel nuclei, correlating with vowel articulation. This suggests that the AAI method captures key articulatory nuances relevant to speech rate.

Key Results

The proposed CDNN-based approaches consistently outperform the baseline Temporal and Selected Sub-band Correlation (TCSSBC) method.

  • Seen Subject Condition: The CDNN approach using estimated articulatory features (CDNNearti) achieved an 81.58% relative improvement in Pearson correlation coefficient (p) compared to the TCSSBC baseline.
  • Unseen Subject Condition: CDNNearti still showed a substantial 73.68% relative improvement over TCSSBC.
  • Comparison of CDNN Variants: CDNNearti generally performed best, followed by CDNNmfcc (acoustic features only), CDNNiarti (interpolated articulograms), and CDNNarti (direct articulograms). This highlights the value of the AAI-derived features.
  • AAI Feature Quality: The AAI method showed moderate correlation coefficients (mean CC of 0.42-0.55 across subjects) for estimating VTTP dimensions, with higher correlations in the tongue region.

Limitations and Future Work

  • Limited Subject Generalizability: The authors acknowledge that the CDNN-based methods' performance is less consistent across subjects in the "unseen subject" condition. This is attributed to the limited number of subjects (four) in the dataset, which hinders the model's generalizability to entirely new speakers.
  • Future Directions:
    • Develop an end-to-end model that directly extracts articulatory features from speech and estimates speech rate.
  • Increase the number of speakers in the training data to improve model generalizability.
  • Investigate performance under various noisy conditions to create a more robust speech rate estimation model.

Summary updated

BibTeX
@inproceedings{mannem2019acoustic,
  author = {Renuka Mannem and Jhansi Mallela and Aravind Illa and Prasanta Kumar Ghosh},
  title = {{Acoustic and Articulatory Feature Based Speech Rate Estimation Using a Convolutional Dense Neural Network}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure