Abstract
We propose a method for zero-resource domain adaptation of DNN acoustic models, for use in low-resource situations where the only in-language training data available may be poorly matched to the intended target domain. Our method uses a multi-lingual model in which several DNN layers are shared between languages. This architecture enables domain adaptation transforms learned for one well-resourced language to be applied to an entirely different low-resource language. First, to develop the technique we use English as a well-resourced language and take Spanish to mimic a low-resource language. Experiments in domain adaptation between the conversational telephone speech (CTS) domain and broadcast news (BN) domain demonstrate a 29% relative WER improvement on Spanish BN test data by using only English adaptation data. Second, we demonstrate the effectiveness of the method for low-resource languages with a poor match to the well-resourced language. Even in this scenario, the proposed method achieves relative WER improvements of 18-27% by using solely English data for domain adaptation. Compared to other related approaches based on multi-task and multi-condition training, the proposed method is able to better exploit well-resource language data for improved acoustic modelling of the low-resource target domain.
Introduction
Automatic Speech Recognition (ASR) models are notoriously sensitive to "domain shift." A model trained on clear, conversational telephone speech (CTS) often performs poorly when suddenly asked to transcribe noisy broadcast news (BN) or topical broadcasts. Standard domain adaptation typically requires at least some transcribed data from the new target domain. However, for many of the world's low-resource (LR) languages, such data simply does not exist.
This paper introduces a method for zero-resource domain adaptation, addressing the challenge of improving ASR systems for languages like Tagalog or Lithuanian in target domains where no in-language training data is available. The proposed solution leverages data from a well-resourced (WR) language, such as English, to adapt the acoustic model of the low-resource language.
Methodology
The core of the approach is a multi-lingual Time Delay Neural Network (TDNN) architecture. The process follows three main steps:
- Multi-lingual Training: The authors train a network where the initial "hidden" layers are shared across languages (e.g., English and Spanish), while the final output layers (senones) are language-specific. This forces the early layers to learn language-independent acoustic features.
- Domain Adaptation: The shared layers of this multi-lingual model are fine-tuned using data from a well-resourced language in the target domain (e.g., English Broadcast News). During this phase, the language-specific output layers are frozen.
- Weight Transfer: The adapted shared layers are then plugged back into the low-resource language's model.
This effectively "transfers" the acoustic transformation learned from English news to the low-resource language, based on the hypothesis that the acoustic difference between domains is somewhat universal across languages.
Experimental Setup
Datasets
- Source Domain (CTS): English and Spanish data from the Fisher corpus (~200 and ~163 hours respectively). For Tagalog and Lithuanian, the IARPA Babel full language packs (80 and 40 hours) were used.
- Target Domain (BN/TB): English and Spanish HUB4 data (~150 and ~30 hours). For Tagalog and Lithuanian, evaluation sets were provided by the IARPA MATERIAL programme.
- Evaluation: Word Error Rate (WER) was measured on standard evaluation sets like the 1997 HUB4 English Evaluation.
Results
The authors first tested the theory using Spanish as a "proxy" for a low-resource language. The baseline system (trained only on Spanish telephone data) had a 40.0% WER on Spanish Broadcast News. By using the proposed cross-lingual adaptation with English news data, the WER dropped to 28.4%—a 29% relative improvement without using a single second of Spanish news for training.
When applied to actual low-resource languages in the MATERIAL task, the method achieved significant gains:
- Tagalog: Achieved an 18.3% relative WER improvement on wideband conditions.
- Lithuanian: Achieved a 27.5% relative WER improvement.
As shown in the table above, the proposed method consistently outperformed other common strategies like multi-task or multi-condition training, proving that explicit adaptation of shared layers is a more effective way to leverage well-resourced data.
Discussion and Insights
- Acoustic Universality: The most striking insight is that the "delta" between domains—the acoustic shift from a telephone line to a news studio—is portable. The network learns how to compensate for channel noise and environmental acoustics in English and can apply that same logic to Spanish or Tagalog.
- Shared Feature Extraction: By forcing the model to learn multiple languages simultaneously, the shared layers are incentivized to ignore language-specific phonetics and focus on robust, fundamental acoustic representations.
- Efficiency: Unlike semi-supervised methods that require expensive decoding on massive amounts of untranscribed data to generate "pseudo-labels," this method is a direct weight transfer, making it much faster to deploy in rapid-response scenarios.
Conclusion
While the method proved effective for Tagalog and Lithuanian, the authors note that its performance might vary depending on the acoustic distance between the WR and LR languages. Future work should explore how this transfer learning holds up in modern End-to-End (E2E) or sequence-trained models. Additionally, there is potential to combine this weight transfer with other techniques like i-vectors or bottleneck features to further enhance performance in zero-resource settings.
Summary updated