A comparative study of BNF and DNN multilingual training on cross-lingual low-resource speech recognition

Haihua Xu, Van Hai Do, Xiong Xiao, Engsiong ChngInterspeech 2015 · DOI

Abstract

Building Automatic Speech Recognition (ASR) systems for low-resource languages remains a significant challenge due to the scarcity of transcribed audio. This paper evaluates two prominent neural network architectures—Multilingual Bottleneck Features (MBNF) and hybrid Multilingual Deep Neural Networks (MDNN)—for cross-lingual transfer. By training on "rich-resource" source languages and adapting to target languages with as little as 3 hours of data, we compare their robustness and fine-tuning potential. Our findings demonstrate that while both methods outperform monolingual baselines, the MBNF approach is more resilient to noise and data scarcity, yielding significantly higher gains from fine-tuning compared to MDNN.

Introduction

The development of robust ASR systems typically necessitates hundreds of hours of high-quality transcribed speech. For the vast majority of the world's languages, such resources are unavailable. Cross-lingual transfer learning offers a solution by leveraging data from multiple source languages to improve performance on a target low-resource language. This study specifically addresses the choice between using neural networks as feature extractors (BNF) versus direct phonetic classifiers (DNN) in these constrained environments.

Methodology

The research compares two primary frameworks for cross-lingual adaptation:

Multilingual DNN (MDNN)

The MDNN utilizes a multi-task learning setup. Several source languages share hidden layers for feature extraction, while maintaining language-specific "softmax" output layers for classification. When adapting to a new language, the shared hidden layers are transferred, and a new output layer is trained.

Multilingual DNN architecture and transfer process

Multilingual Bottleneck (MBNF)

The MBNF approach employs a "stacked" architecture. A first network is trained to produce a low-dimensional "bottleneck" layer. These features are then fed into a second network, with the final output serving as input features for a traditional GMM-HMM system. This allows the neural network to act as a sophisticated feature extractor rather than a direct classifier.

Fine-tuning Strategies

The authors explored updating pre-trained models with target data. For MDNNs, layer-wise tuning was employed to mitigate "catastrophic forgetting" of the general features learned from source languages. For MBNFs, the bottleneck networks themselves were fine-tuned to better capture target-specific acoustic characteristics.

Experimental Setup

Data was sourced from the IARPA Babel project, focusing on conversational speech across diverse linguistic families.

  • Source Languages (High Resource): Cantonese, Pashto, Tagalog, and Turkish (~80-140 hours each).
  • Target Languages (Low Resource): Vietnamese and Tamil.
  • Data Tiers: Limited Language Pack (LLP, ~11h) and Very Limited Language Pack (VLLP, ~3h).
  • Conditions: The Tamil dataset provided a higher noise floor, testing the robustness of the architectures under non-ideal recording conditions.

Results and Discussion

Both multilingual approaches significantly improved over monolingual baselines, with relative Word Error Rate (WER) reductions of 6.1–8.6% on the 3-hour VLLP sets. However, distinct differences emerged in their adaptation capabilities.

The Fine-Tuning Advantage

A key differentiator was the effectiveness of fine-tuning. Fine-tuning the MBNF network resulted in absolute WER reductions of up to 2.8%. In contrast, MDNN fine-tuning yielded marginal improvements (approx. 0.4%) and struggled in noisy conditions.

Frame accuracy comparison during fine-tuning for MBNF and MDNN variants

Robustness to Noise and Data Scarcity

The MBNF system proved more stable. On the 3-hour Vietnamese set, MBNF improved to 67.2% WER after tuning, compared to 69.0% for the MDNN. In the noisy Tamil VLLP task, MBNF reached 77.1% WER, while MDNN performance often stagnated or degraded during tuning, suggesting that hybrid classifiers are more prone to overfitting on tiny, noisy datasets.

Conclusion

For extremely low-resource ASR (under 10 hours of data), the Multilingual Bottleneck approach is superior to the hybrid MDNN. MBNF's ability to learn universal, robust features that are highly adaptable through fine-tuning makes it the preferred choice for challenging, data-scarce environments. Future work will investigate language selection strategies to further optimize cross-lingual transfer by selecting source languages linguistically closer to the target.

Summary updated

BibTeX
@inproceedings{xu2015comparative,
  author = {Haihua Xu and Van Hai Do and Xiong Xiao and Engsiong Chng},
  title = {{A comparative study of BNF and DNN multilingual training on cross-lingual low-resource speech recognition}},
  booktitle = {Interspeech},
  year = {2015},
  doi = {10.21437/interspeech.2015-481},
}
Figure