Automatic Speech Recognition with Articulatory Information and a Unified Dictionary for Hindi, Marathi, Bengali and Oriya

Debadatta Dash, Myungjong Kim, Kristin Teplansky, Jun Wang2018 · DOI

TL;DR

This paper introduces a novel approach to Automatic Speech Recognition (ASR) for low-resource Indian languages by integrating articulatory information (tongue and lip movements) with acoustic features. They demonstrate that this multimodal approach, combined with a unified dictionary of phonemes across Hindi, Marathi, Bengali, and Oriya, significantly improves recognition accuracy, especially for speaker-independent tasks.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems have seen tremendous progress for widely spoken languages like English and Mandarin. However, for the vast linguistic landscape of India, with thousands of languages and a significant rural population, ASR technology remains in its nascent stages. This is primarily due to a lack of sufficient data and resources for training robust models. Existing research on Indian languages has largely focused on Hindi, with limited work on others like Marathi, Bengali, and Oriya. This paper aims to bridge this gap by exploring how to enhance ASR for these languages, particularly by leveraging articulatory information, which has shown promise in other languages but has been under-explored for Indian languages.

Methods

The researchers employed a multi-modal ASR approach, combining acoustic features with articulatory motion data.

  • Articulatory Motion Tracking: They used an electromagnetic articulograph (Wave system) to record the 3D movements of the tongue tip (TT), tongue back (TB), upper lip (UL), and lower lip (LL) during speech production. Head motion was corrected using a forehead sensor.
  • Feature Extraction: Acoustic features were represented by 39-dimensional Mel-Frequency Cepstral Coefficients (MFCCs) with their derivatives. Articulatory features comprised 24 dimensions (8 static values from 4 sensors and their first and second derivatives). These were concatenated to form a combined feature vector.
  • ASR Models: Three ASR models were used:
  • Gaussian Mixture Model-Hidden Markov Model (GMM-HMM): A traditional ASR approach.
  • Deep Neural Network-Hidden Markov Model (DNN-HMM): A more advanced deep learning model.
  • Long Short-Term Memory-Hidden Markov Model (LSTM-HMM): A recurrent neural network architecture well-suited for sequential data like speech.
  • Speaker-Dependent (SD) and Speaker-Independent (SI) Experiments: They conducted experiments for both scenarios. For SI, Procrustes matching was used to normalize articulatory data across speakers, mitigating inter-talker physiological differences.
  • Unified Dictionary (UD): To address the challenges of multi-lingual ASR for Indian languages, they created a unified dictionary by combining common and language-specific phonemes from Hindi, Marathi, Bengali, and Oriya. This dictionary contained 95 phonemes (41 common, 54 unique).

Datasets

The study utilized data collected from two native male speakers (average age 27.5 years). Each speaker uttered 132 phrases in each of the four languages (Hindi, Marathi, Bengali, and Oriya). This resulted in approximately 5,534 phonemes and 612 words per language. The number of unique phonemes per language were: Hindi (55), Marathi (66), Bengali (51), and Oriya (46). The average number of unique words across the four languages was 412. No new benchmark datasets were introduced; rather, existing data collection methodologies were adapted.

Key Insights and Ideas

  • Articulatory Information is Complementary: Integrating articulatory movements (tongue and lip) with acoustic features significantly improves ASR performance for Indian languages. This highlights that physical speech production provides valuable information not fully captured by acoustics alone.
  • Cross-Lingual Phonetic Similarity: A notable finding is the high degree of phonetic similarity between Hindi and Marathi, and similarly between Bengali and Oriya, both in acoustic and articulatory domains. This observation is visually supported by similar motion contours for these language pairs.
  • Unified Dictionary for Multilingual ASR: The creation of a unified dictionary, encompassing common and unique phonemes, proved highly effective for multilingual ASR. This approach leverages the shared linguistic heritage of Indian languages (originating from Brahmi script) to build a more robust system.
  • LSTM's Superiority for Low-Resource ASR: LSTM models demonstrated superior performance compared to GMM-HMM and DNN-HMM, especially in speaker-independent tasks and with limited data. This suggests LSTM's suitability for low-resource language ASR.
  • Articulatory Data Alone is Insufficient: While articulatory data is beneficial, using it in isolation resulted in poorer performance than using acoustic features alone, underscoring the need for a multimodal approach.

Key Results

  • Speaker-Dependent ASR: Adding articulatory information improved ASR performance (reduced Phoneme Error Rate - PER) for all four languages across GMM-HMM and DNN-HMM models. The average PER reduction was around 3% with DNN-HMM compared to GMM-HMM.
  • Speaker-Independent ASR:
    • Procrustes matching for articulatory normalization yielded an average 5% PER improvement.
  • The inclusion of articulatory features further enhanced SI performance by an average of 6% PER reduction.
  • LSTM models generally outperformed GMM and DNN for SI tasks.
  • Unified Dictionary Performance: The multi-lingual ASR with a Unified Dictionary achieved significantly reduced PERs, around 16%, for cross-lingual testing. This was attributed to the increased effective training data by combining phonemes from all four languages.
  • Cross-Lingual Testing (Table 2): Lower PERs were observed for Hindi-Marathi (e.g., 18.2% Hindi trained on Hindi, 42.3% Hindi trained on Marathi) and Bengali-Oriya pairs, confirming their phonetic similarity.

Limitations and Future Work

  • Limited Data Size: The study used data from only two male speakers. The authors acknowledge that the data size is small and no female speakers were included, making it unclear if the findings generalize to a larger, more diverse population.
  • Future Work:
    • Generalization: Expanding the dataset to include more speakers, including females, is crucial for validating the findings.
  • Practical Articulatory Data: Due to the logistic difficulties of collecting direct articulatory data, the authors suggest exploring the use of quasi-articulatory data derived from acoustic signals via speaker-independent inverse mapping.
  • Broader Language Coverage: Extending the multilingual approach to a wider range of Indian languages is a promising direction.

Summary updated

BibTeX
@misc{dash2018automatic,
  author = {Debadatta Dash and Myungjong Kim and Kristin Teplansky and Jun Wang},
  title = {{Automatic Speech Recognition with Articulatory Information and a Unified Dictionary for Hindi, Marathi, Bengali and Oriya}},
  year = {2018},
  doi = {10.21437/interspeech.2018-2122},
}
Figure