Abstract
Recently there has been some interest in the question of how to build LVCSR systems for the low-resource languages. The scenario we focus on here is having only one hour of acoustic training data in the "target" language, but more plentiful data in other languages. This paper presents approaches using MLP based features: we construct a low-resource system with additional sources of information from the non-target languages to train the cross-lingual MLPs. A hierarchical architecture and multi-stream strategy are applied on the cross-lingual phone level, to improve the neural network more discriminatively. Additionally, an elaborate ensemble system with various acoustic feature streams and context expansion lengths is proposed. After system combination with these two strategies we get significant improvements of more than 8% absolute versus a conventional baseline in this low-resource scenario with only one hour of target training data.
Introduction
Modern Automatic Speech Recognition (ASR) systems typically require hundreds or thousands of hours of transcribed speech to achieve high performance. However, for the vast majority of the world's languages, such massive datasets are unavailable. This research addresses an extreme "low-resource" scenario where only one hour of acoustic training data is available for the target language. To overcome the limitations of such sparse data, we propose leveraging phonetic knowledge from data-rich languages and maximizing information extraction through an ensemble of diverse acoustic features.
Methodology
The proposed system utilizes Multi-Layer Perceptrons (MLPs) to generate "posterior features"—probabilities of specific phones being spoken. Two primary strategies are employed:
Cross-Lingual Hierarchical Training
This strategy leverages "source" languages with more abundant data (German and Spanish). We employ a hierarchical architecture where the outputs of source-language MLPs are fed into a second stage. This second stage is refined using the limited target language data via a State Time Alignment (STA) algorithm to map phonetic similarities. A multi-stream merger then combines outputs from German, Spanish, and target-trained networks into a robust feature set.
Ensemble MLP Strategy
To ensure robustness, we create an ensemble of networks trained on the target language using diverse acoustic representations, including Perceptual Linear Prediction (PLP), Mel-frequency Cepstral Coefficients (MFCC), and Tempo-RAl Patterns (TRAP). By varying the context window lengths (7, 9, or 11 frames), we capture a wider range of phonetic cues. These diverse models are merged to provide a comprehensive feature set for the recognizer.
Both strategies are integrated into a Tandem approach, where MLP outputs are processed via Principal Component Analysis (PCA) and fed into a traditional HMM-GMM recognizer.
Experimental Setup
The target language is Callhome English, using a randomly selected 1-hour subset for training. Source languages include Callhome German (15 hours) and Callhome Spanish (16 hours). Evaluation was performed on a 1.8-hour test set from the Callhome English corpus using a 62K word dictionary.
Results and Discussion
The performance was measured using Word Error Rate (WER). The conventional PLP-HMM-GMM baseline achieved a 72.57% WER. Implementing the cross-lingual strategy alone reduced the WER to 66.17%, while the ensemble strategy alone reached 66.62%. The combination of both strategies yielded the best performance at 64.45% WER.
This represents an 8.12% absolute improvement over the baseline. The results suggest that phonetic universality allows MLPs to capture cross-lingual features effectively, and that diversity in acoustic features is more beneficial than depth when data is scarce.
Conclusion
We demonstrated that combining cross-lingual knowledge transfer with an ensemble of diverse acoustic features significantly improves ASR performance in low-resource scenarios. While a 64.45% WER remains high for conversational speech, the 11.2% relative gain over the baseline is substantial. Future work will explore combining these MLP-based features with model-level innovations like Subspace Gaussian Mixture Models (SGMMs) and investigating scalability with slightly larger datasets.
Summary updated