Cross-Lingual and Ensemble MLPs Strategies for Low-Resource Speech Recognition

Yanmin Qian, Jia LiuInterspeech 2012 · OpenReview

Abstract

Recently there has been some interest in the question of how to build LVCSR systems for the low-resource languages. The scenario we focus on here is having only one hour of acoustic training data in the "target" language, but more plentiful data in other languages. This paper presents approaches using MLP based features: we construct a low-resource system with additional sources of information from the non-target languages to train the cross-lingual MLPs. A hierarchical architecture and multi-stream strategy are applied on the cross-lingual phone level, to improve the neural network more discriminatively. Additionally, an elaborate ensemble system with various acoustic feature streams and context expansion lengths is proposed. After system combination with these two strategies we get significant improvements of more than 8% absolute versus a conventional baseline in this low-resource scenario with only one hour of target training data.

Introduction

Modern Automatic Speech Recognition (ASR) systems typically require hundreds or thousands of hours of transcribed speech to achieve high performance. However, for the vast majority of the world's languages, such massive datasets are unavailable. This research addresses an extreme "low-resource" scenario where only one hour of acoustic training data is available for the target language. To overcome the limitations of such sparse data, we propose leveraging phonetic knowledge from data-rich languages and maximizing information extraction through an ensemble of diverse acoustic features.

Methodology

The proposed system utilizes Multi-Layer Perceptrons (MLPs) to generate "posterior features"—probabilities of specific phones being spoken. Two primary strategies are employed:

Cross-Lingual Hierarchical Training

This strategy leverages "source" languages with more abundant data (German and Spanish). We employ a hierarchical architecture where the outputs of source-language MLPs are fed into a second stage. This second stage is refined using the limited target language data via a State Time Alignment (STA) algorithm to map phonetic similarities. A multi-stream merger then combines outputs from German, Spanish, and target-trained networks into a robust feature set.

Architecture of the cross-lingual hierarchical MLP system.

Ensemble MLP Strategy

To ensure robustness, we create an ensemble of networks trained on the target language using diverse acoustic representations, including Perceptual Linear Prediction (PLP), Mel-frequency Cepstral Coefficients (MFCC), and Tempo-RAl Patterns (TRAP). By varying the context window lengths (7, 9, or 11 frames), we capture a wider range of phonetic cues. These diverse models are merged to provide a comprehensive feature set for the recognizer.

Architecture of the ensemble MLP strategy using multiple acoustic features.

Both strategies are integrated into a Tandem approach, where MLP outputs are processed via Principal Component Analysis (PCA) and fed into a traditional HMM-GMM recognizer.

Experimental Setup

The target language is Callhome English, using a randomly selected 1-hour subset for training. Source languages include Callhome German (15 hours) and Callhome Spanish (16 hours). Evaluation was performed on a 1.8-hour test set from the Callhome English corpus using a 62K word dictionary.

Results and Discussion

The performance was measured using Word Error Rate (WER). The conventional PLP-HMM-GMM baseline achieved a 72.57% WER. Implementing the cross-lingual strategy alone reduced the WER to 66.17%, while the ensemble strategy alone reached 66.62%. The combination of both strategies yielded the best performance at 64.45% WER.

This represents an 8.12% absolute improvement over the baseline. The results suggest that phonetic universality allows MLPs to capture cross-lingual features effectively, and that diversity in acoustic features is more beneficial than depth when data is scarce.

Conclusion

We demonstrated that combining cross-lingual knowledge transfer with an ensemble of diverse acoustic features significantly improves ASR performance in low-resource scenarios. While a 64.45% WER remains high for conversational speech, the 11.2% relative gain over the baseline is substantial. Future work will explore combining these MLP-based features with model-level innovations like Subspace Gaussian Mixture Models (SGMMs) and investigating scalability with slightly larger datasets.

Summary updated

Abstract

Recently there has been some interest in the question of how to build LVCSR systems for the low-resource languages. The scenario we focus on here is having only one hour of acoustic training data in the "target" language, but more plentiful data in other languages. This paper presents approaches using MLP based features: we construct a low-resource system with additional sources of information from the non-target languages to train the cross-lingual MLPs. A hierarchical architecture and multi-stream strategy are applied on the cross-lingual phone level, to improve the neural network more discriminatively. Additionally, an elaborate ensemble system with various acoustic feature streams and context expansion lengths is proposed. After system combination with these two strategies we get significant improvements of more than 8% absolute versus a conventional baseline in this low-resource scenario with only one hour of target training data.
BibTeX
@inproceedings{qian2012cross,
  author = {Yanmin Qian and Jia Liu},
  title = {{Cross-Lingual and Ensemble MLPs Strategies for Low-Resource Speech Recognition}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=P7hHCPEMuZ},
}
Figure