Using resources from a closely-related language to develop ASR for a very under-resourced language: a case study for iban

Sarah Samson Juan, Laurent Besacier, François Portet, Mohamed Dyab2015 · DOI

TL;DR

This paper tackles the challenge of building Automatic Speech Recognition (ASR) systems for under-resourced languages like Iban, which suffer from a severe lack of data. The authors demonstrate that leveraging resources from a closely-related, well-resourced language (Malay) significantly improves ASR performance, both for creating pronunciation dictionaries and for training acoustic models. They also introduce a promising "zero-shot" ASR approach that can generate transcripts for Iban using a Malay ASR system.

What Problem Does This Paper Address?

Developing robust ASR systems typically requires vast amounts of transcribed speech data for training acoustic and language models. However, many of the world's languages, like Iban, are "under-resourced," meaning they have very limited or no readily available transcribed speech, pronunciation dictionaries, or even written materials. This scarcity makes it incredibly difficult to build effective ASR systems for these languages using standard methods. The paper aims to bridge this gap by exploring how to effectively utilize resources from a related, well-resourced language to overcome these data limitations.

Methods

The researchers propose several strategies to build an Iban ASR system by exploiting resources from Malay:

  • Semi-supervised Pronunciation Dictionary Construction: Instead of building an Iban pronunciation dictionary from scratch, they use a semi-supervised approach. This involves training a grapheme-to-phoneme (G2P) model on Malay data and then using it to predict pronunciations for Iban words. This initial prediction is then refined by a native speaker, and the corrected data is used to train a better Iban-specific G2P model. They also explore a "Hybrid G2P" approach that combines Malay and Iban G2P models.
  • Cross-lingual Acoustic Model Training: They investigate two main ASR modeling techniques: Subspace Gaussian Mixture Models (SGMM) and Deep Neural Networks (DNNs). For both, they explore "cross-lingual" training, where models are initialized or trained using data from Malay (a closely-related language) or English (a non-closely-related language). This includes using Malay or English Universal Background Models (UBMs) for SGMMs and transferring learned hidden layers from Malay or English DNNs.
  • Zero-Shot ASR: As a novel approach, they propose a "zero-shot" ASR system. This method involves using a pre-trained Malay ASR system to automatically transcribe Iban speech data. These automatically generated transcripts, though potentially noisy, are then used as training data for a new Iban ASR system (GMM, SGMM, or DNN).

Datasets

  • Iban Data:
    • Approximately 8 hours of clean speech (split into 7 hours for training and 1 hour for testing).
  • A text corpus of 2 million words from online news articles, normalized to create a vocabulary of 37,000 unique words.
  • No initial pronunciation dictionary was available.
  • Malay Data:
    • The MASS corpus, containing about 140 hours of clean, read Malay speech. This was used for training Malay ASR systems and G2P models.
  • English Data:
    • The TED-LIUM corpus (first release), containing about 120 hours of speech from TED talks. This was used as a non-closely-related language resource for comparison.

Key Insights and Ideas

  • The Power of "Closeness": The most significant finding is that using resources from a closely-related language (Malay) is demonstrably more effective than using resources from a distant language (English) for improving ASR performance in an under-resourced language like Iban. This applies to both pronunciation dictionary creation and acoustic model training.
  • Bridging the Lexicon Gap: The semi-supervised G2P approach, especially the "Hybrid G2P" method, significantly reduces the effort required to build a functional pronunciation dictionary for Iban, proving that related language data can bootstrap this crucial component.
  • Cross-lingual Transfer Works: Cross-lingual training, particularly with Malay data, provides substantial improvements in Word Error Rate (WER) for Iban ASR, especially when training data is extremely limited (e.g., 1 hour).
  • DNNs Benefit from Transfer: Transferring learned hidden layers from Malay DNNs to an Iban DNN system also yields performance gains, highlighting the generalizability of learned acoustic features.
  • Zero-Shot ASR is Promising: The zero-shot approach, which uses automatically generated transcripts from a related language's ASR system, achieves performance surprisingly close to supervised training, offering a viable path for ASR development when even limited transcribed data is unavailable.

Key Results

  • Pronunciation Dictionary: The "Hybrid G2P" (combining Malay and Iban G2P) and the Malay G2P alone provided the best results for Iban ASR, outperforming the English G2P and a purely grapheme-based approach. Speaker adaptation further improved these results.
  • Cross-lingual SGMM: Using a Malay UBM for cross-lingual SGMM training resulted in a 9.5% absolute WER improvement for Iban ASR with only 1 hour of training data, compared to a 7% improvement with an English UBM.
  • Cross-lingual DNN: Fine-tuning hidden layers from a Malay DNN improved Iban ASR performance, especially for the 1-hour training condition. A speaker-adapted Iban DNN trained monolingually achieved 15.8% WER with 7 hours of data.
  • Zero-Shot ASR: The unsupervised (zero-shot) system achieved performance only 2-3% worse than supervised systems, demonstrating its potential. Interestingly, SGMM and DNN showed similar performance in the zero-shot setting, suggesting DNNs might be more sensitive to noisy transcripts.

Limitations and Future Work

  • Limitations:
    • The zero-shot DNN performance was not significantly better than SGMM, possibly due to sensitivity to noisy automatically generated transcripts.
  • The paper notes that merging DNNs with different feature transforms did not yield improvements, suggesting careful handling of feature spaces is needed.
  • Future Work:
    • The authors suggest using confidence measures to better select hypotheses for DNN training in the zero-shot scenario.
  • Further exploration of multilingual UBMs and DNNs could yield additional benefits.
  • The paper highlights the availability of their Iban ASR corpus and Kaldi scripts on GitHub, encouraging further research in this area.

Summary updated

Abstract

This paper presents our strategies for developing an automatic speech recognition system for Iban, an under-resourced language. We faced several challenges such as no pronunciation dictionary and lack of training material for building acoustic models. To overcome these problems, we proposed approaches which exploit resources from a closely-related language (Malay). We developed a semi-supervised method for building the pronunciation dictionary and applied cross-lingual strategies for improving acoustic models trained with very limited training data. Both approaches displayed very encouraging results, which show that data from a closely-related language, if available, can be exploited to build ASR for a new language. In the final part of the paper, we present a zero-shot ASR using Malay resources that can be used as an alternative method for transcribing Iban speech.
BibTeX
@misc{juan2015using,
  author = {Sarah Samson Juan and Laurent Besacier and François Portet and Mohamed Dyab},
  title = {{Using resources from a closely-related language to develop ASR for a very under-resourced language: a case study for iban}},
  year = {2015},
  doi = {10.21437/interspeech.2015-318},
}
Figure