Multilingual bottle-neck features and its application for under-resourced languages

Ngoc Thang Vu, Florian Metze, Tanja Schultz

TL;DR

This paper introduces "multilingual bottle-neck features" and an "open target language" MLP approach to significantly improve speech recognition for under-resourced languages. By leveraging knowledge from multiple languages, they achieve substantial reductions in word error rates, especially when training data is scarce.

What Problem Does This Paper Address?

The field of speech recognition has made incredible strides, but most of this progress is concentrated on a handful of well-resourced languages. The vast majority of the world's languages remain "under-resourced," meaning there's a significant challenge in porting speech processing systems to them efficiently and cost-effectively. Traditional approaches often require extensive manual effort and large amounts of language-specific data, which simply isn't available for many languages. This paper tackles the problem of rapid language adaptation for these under-resourced languages, aiming to make speech recognition more accessible globally.

Methods

The core innovation lies in two main areas:

  • Multilingual Bottle-Neck (BN) Features: Instead of training a speech recognition model from scratch for each new language, the authors propose using a "bottle-neck" feature extraction method. This involves training a Multi-Layer Perceptron (MLP) on data from multiple languages simultaneously. The MLP learns to extract a compressed, highly informative representation (the "bottle-neck" features) that captures commonalities across languages. These features are then used as input for a subsequent acoustic model.
  • "Open Target Language" MLP: A key challenge in multilingual training is that the universal phone set might not cover all the specific phonemes of a new language. To address this, they introduce an "open target language" MLP. This strategy allows the MLP to be trained with a more flexible target set, effectively expanding its ability to represent phonemes from various languages, even those not perfectly aligned with the initial multilingual training. This is particularly useful when adapting to a new language with limited data.

The process involves:

  1. Pooling phoneme sets from multiple languages (English, French, German, Spanish) to create a universal phone set.
  2. Training a multilingual MLP (ML-MLP) on this pooled data.
  3. For a new language (Vietnamese), identifying phonemes not covered by the universal set.
  4. Using the ML-MLP's learned features as initialization for training a new MLP for the target language, or directly adapting the ML-MLP itself using the "open target language" strategy.

Datasets

The primary dataset used is the GlobalPhone corpus. This corpus contains speech data from 20 languages, with the authors specifically selecting:

  • Vietnamese
  • English
  • French
  • German
  • Spanish

For language model building, they utilized a rapid crawling process to gather large text corpora.

Key Insights and Ideas

  • Multilingual Initialization is Powerful: Initializing an MLP for a new language with weights learned from a multilingual MLP significantly improves performance compared to random initialization. This suggests that shared acoustic knowledge across languages is highly beneficial.
  • Faster Training: Using multilingual MLP initialization not only improves accuracy but also reduces training time by a notable amount (around 40% in their experiments).
  • "Open Target Language" for Flexibility: The proposed "open target language" MLP strategy is crucial for handling under-resourced languages where the universal phone set might be incomplete. It allows for more flexible model training, especially when dealing with limited data.
  • Robustness: Systems initialized with multilingual MLPs are more robust, leading to better performance across various languages.

Key Results

The paper reports significant improvements in Syllable Error Rate (SyllER) for Vietnamese ASR:

  • With 22.5 hours of training data: A 15.8% relative improvement in SyllER was achieved using the multilingual bottle-neck features.
  • With only 2 hours of training data: A 16.9% relative improvement in SyllER was observed.

In general experiments:

  • Multilingual MLP initialization consistently outperformed random initialization for frame-wise classification accuracy across English, French, German, and Spanish.
  • ASR systems using multilingual MLP features outperformed baseline MFCC systems, with relative improvements in WER of up to 10%.

Limitations and Future Work

  • Limited Universal Phone Set Coverage: The authors acknowledge that even their universal phone set covers only about 30% of IPA symbols, necessitating the "open target language" approach for languages with many unique phonemes.
  • Future Work: The paper implicitly suggests that further exploration of the "open target language" MLP and its application to other under-resourced languages would be valuable. The success with Vietnamese indicates a promising direction for broader language coverage.

Summary updated

Abstract

In this paper we present our latest investigation on multilingual bottle-neck (BN) features and its application to rapid language adaptation to new languages. We show that the overall performance of a Multilayer Perceptron (MLP) network improves significantly by initializing it with a multilingualMLP. Furthermore, ASR performance increases on both, on those languages which were used for multilingual MLP training, and on a new language. We propose a new strategy called “open target language” MLP to train more flexible models for language adaptation, which is particularly suited for small amounts of training data. The final results on the Vietnamese GlobalPhone database gave 15.8% relative improvement in terms of Syllable Error Rate (SyllER) for the ASR system trained with 22.5h data and 16.9% relative gains for the system trained with only 2h data.
BibTeX
@inproceedings{vu2012multilingual,
  author = {Ngoc Thang Vu and Florian Metze and Tanja Schultz},
  title = {{Multilingual bottle-neck features and its application for under-resourced languages}},
  booktitle = {Sltu},
  year = {2012},
}
Figure