2016 BUT Babel System: Multilingual BLSTM Acoustic Model with i-Vector Based Adaptation

Martin Karafiát, Murali Karthick Baskar, Pavel Matějka, Karel Veselý, František Grézl, Lukáš Burget, Jan ČernockýInterspeech 2017

TL;DR

This paper tackles the challenge of building effective Automatic Speech Recognition (ASR) systems for low-resource languages, a common problem in real-world applications. The authors demonstrate that multilingual training of acoustic models and feature extractors significantly improves performance, and they explore novel ways to integrate i-vector based speaker adaptation into BLSTM models, achieving further gains.

What Problem Does This Paper Address?

Developing ASR systems for languages with limited transcribed data is a major hurdle. Traditional approaches often require extensive language-specific training data, which is simply unavailable for many of the world's languages. The IARPA Babel program, which this work is based on, specifically aimed to address this by focusing on rapid development of ASR for numerous low-resource languages. The core problem is how to leverage existing resources and knowledge from high-resource languages to build effective systems for these data-scarce scenarios. This involves finding ways to share information across languages, both in how speech features are extracted and how acoustic models are trained, and also how to adapt these models to individual speakers efficiently.

Methods

The researchers employed several key techniques:

  • Multilingual Training: Instead of training separate models for each language, they trained models on data from multiple languages simultaneously. This was applied to both:
  • Feature Extraction: Using a Stacked Bottle-Neck (SBN) network trained on a diverse set of languages to create general-purpose speech features.
  • Acoustic Modeling: Training Bidirectional Long Short-Term Memory (BLSTM) recurrent neural networks on data from multiple languages. A "block-softmax" output layer was used, with per-language softmaxes, and then fine-tuned for the target language.
  • i-Vector Based Speaker Adaptation: i-vectors are compact representations of speaker characteristics. The paper explores two methods for integrating them into BLSTM models:
  • Augmentation: Appending the i-vector to the output of specific BLSTM layers.
  • SAT-DNN (Speaker Adaptive Training - Deep Neural Network): Training a small auxiliary network (ivec-NN) to transform the i-vector and then adding its output to the input or intermediate layers of the main BLSTM acoustic model. This approach was found to be more effective.
  • Feature Types: They compared different acoustic features, including log-Mel-filter-bank outputs and bottleneck features derived from SBN networks.

Datasets

The study utilized data from the IARPA Babel program, which includes conversational telephone speech and other recording types for 24 low-resource languages across four years of the program. They also incorporated non-Babel datasets (Levantine Arabic, English, Mandarin, Spanish) for pre-training. Experiments were evaluated on three specific Year 4 languages: Javanese (JA), Pashto (PA), and Amharic (AM).

Key Insights and Ideas

  • Multilingual Pre-training is Crucial: Training acoustic models and feature extractors on a large, diverse set of languages provides a significant performance boost for low-resource languages, even when the final model is fine-tuned on limited target-language data.
  • BLSTMs Benefit from Multilingual Training: While bottleneck features were beneficial for feed-forward DNNs, BLSTM models performed better with simpler log-Mel-filter-bank features when trained multilingually.
  • i-Vectors Integrate Well with BLSTMs: The paper demonstrates that i-vectors can be effectively used to adapt BLSTM models, a novel contribution as previous work primarily focused on feed-forward DNNs.
  • Middle Layers are Key for i-Vector Adaptation: For BLSTM adaptation, integrating i-vectors into the middle layers of the network (rather than just the input features, as is common with DNNs) yielded better results.

Key Results

  • WER Reductions from Multilingual Pre-training: Over 3% absolute Word Error Rate (WER) reductions were observed on three Babel Year 4 languages due to multilingual pre-training of acoustic models.
  • i-Vector Adaptation Gains: Speaker adaptation using i-vectors resulted in additional WER reductions of over 1% absolute.
  • SAT-DNN Superiority: The SAT-DNN approach for i-vector integration outperformed simple augmentation, particularly when applied to the middle layers of the BLSTM.
  • Feature Choice Matters: For multilingual BLSTMs, basic log-Mel-filter-bank features with pitch information proved more effective than bottleneck features.

Limitations and Future Work

  • Language Diversity vs. Data Amount: While multilingual training was beneficial, the study found that increasing language diversity had a greater impact than simply increasing the amount of data from a few languages. Adding large non-Babel resources did not yield significant improvements, suggesting a saturation point for diversity.
  • i-Vector Integration for DNNs: The paper notes that while SAT-DNN was effective for BLSTMs, it's not clear what the optimal integration strategy is for i-vectors in BLSTM models compared to feed-forward DNNs.
  • Future Work: The authors suggest further exploration into optimal i-vector integration strategies for BLSTM models and potentially investigating other adaptation techniques.

Summary updated

BibTeX
@inproceedings{karafiat2017but,
  author = {Martin Karafiát and Murali Karthick Baskar and Pavel Matějka and Karel Veselý and František Grézl and Lukáš Burget and Jan Černocký},
  title = {{2016 BUT Babel System: Multilingual BLSTM Acoustic Model with i-Vector Based Adaptation}},
  booktitle = {Interspeech},
  year = {2017},
}
Figure