Cross Lingual Transfer Learning for Zero-Resource Domain Adaptation

Alberto Abad, Peter Bell, Andrea Carmantini, Steve RenalsICASSP 2020 · OpenReview

Abstract

We propose a method for zero-resource domain adaptation of DNN acoustic models, for use in low-resource situations where the only in-language training data available may be poorly matched to the intended target domain. Our method uses a multi-lingual model in which several DNN layers are shared between languages. This architecture enables domain adaptation transforms learned for one well-resourced language to be applied to an entirely different low-resource language. First, to develop the technique we use English as a well-resourced language and take Spanish to mimic a low-resource language. Experiments in domain adaptation between the conversational telephone speech (CTS) domain and broadcast news (BN) domain demonstrate a 29% relative WER improvement on Spanish BN test data by using only English adaptation data. Second, we demonstrate the effectiveness of the method for low-resource languages with a poor match to the well-resourced language. Even in this scenario, the proposed method achieves relative WER improvements of 18-27% by using solely English data for domain adaptation. Compared to other related approaches based on multi-task and multi-condition training, the proposed method is able to better exploit well-resource language data for improved acoustic modelling of the low-resource target domain.

Introduction

Automatic Speech Recognition (ASR) models are notoriously sensitive to "domain shift." A model trained on clear, conversational telephone speech (CTS) often performs poorly when suddenly asked to transcribe noisy broadcast news (BN) or topical broadcasts. Standard domain adaptation typically requires at least some transcribed data from the new target domain. However, for many of the world's low-resource (LR) languages, such data simply does not exist.

This paper introduces a method for zero-resource domain adaptation, addressing the challenge of improving ASR systems for languages like Tagalog or Lithuanian in target domains where no in-language training data is available. The proposed solution leverages data from a well-resourced (WR) language, such as English, to adapt the acoustic model of the low-resource language.

Methodology

The core of the approach is a multi-lingual Time Delay Neural Network (TDNN) architecture. The process follows three main steps:

  1. Multi-lingual Training: The authors train a network where the initial "hidden" layers are shared across languages (e.g., English and Spanish), while the final output layers (senones) are language-specific. This forces the early layers to learn language-independent acoustic features.
  2. Domain Adaptation: The shared layers of this multi-lingual model are fine-tuned using data from a well-resourced language in the target domain (e.g., English Broadcast News). During this phase, the language-specific output layers are frozen.
  3. Weight Transfer: The adapted shared layers are then plugged back into the low-resource language's model.
The three-stage process for cross-lingual domain adaptation: multi-lingual training, domain adaptation on well-resourced data, and weight transfer to the low-resource model.

This effectively "transfers" the acoustic transformation learned from English news to the low-resource language, based on the hypothesis that the acoustic difference between domains is somewhat universal across languages.

Experimental Setup

Datasets

  • Source Domain (CTS): English and Spanish data from the Fisher corpus (~200 and ~163 hours respectively). For Tagalog and Lithuanian, the IARPA Babel full language packs (80 and 40 hours) were used.
  • Target Domain (BN/TB): English and Spanish HUB4 data (~150 and ~30 hours). For Tagalog and Lithuanian, evaluation sets were provided by the IARPA MATERIAL programme.
  • Evaluation: Word Error Rate (WER) was measured on standard evaluation sets like the 1997 HUB4 English Evaluation.

Results

The authors first tested the theory using Spanish as a "proxy" for a low-resource language. The baseline system (trained only on Spanish telephone data) had a 40.0% WER on Spanish Broadcast News. By using the proposed cross-lingual adaptation with English news data, the WER dropped to 28.4%—a 29% relative improvement without using a single second of Spanish news for training.

When applied to actual low-resource languages in the MATERIAL task, the method achieved significant gains:

  • Tagalog: Achieved an 18.3% relative WER improvement on wideband conditions.
  • Lithuanian: Achieved a 27.5% relative WER improvement.
WER results for Tagalog and Lithuanian, comparing the baseline mono-lingual system (1), a standard multi-lingual system (2), and the proposed cross-lingual adaptation method (3).

As shown in the table above, the proposed method consistently outperformed other common strategies like multi-task or multi-condition training, proving that explicit adaptation of shared layers is a more effective way to leverage well-resourced data.

Discussion and Insights

  • Acoustic Universality: The most striking insight is that the "delta" between domains—the acoustic shift from a telephone line to a news studio—is portable. The network learns how to compensate for channel noise and environmental acoustics in English and can apply that same logic to Spanish or Tagalog.
  • Shared Feature Extraction: By forcing the model to learn multiple languages simultaneously, the shared layers are incentivized to ignore language-specific phonetics and focus on robust, fundamental acoustic representations.
  • Efficiency: Unlike semi-supervised methods that require expensive decoding on massive amounts of untranscribed data to generate "pseudo-labels," this method is a direct weight transfer, making it much faster to deploy in rapid-response scenarios.

Conclusion

While the method proved effective for Tagalog and Lithuanian, the authors note that its performance might vary depending on the acoustic distance between the WR and LR languages. Future work should explore how this transfer learning holds up in modern End-to-End (E2E) or sequence-trained models. Additionally, there is potential to combine this weight transfer with other techniques like i-vectors or bottleneck features to further enhance performance in zero-resource settings.

Summary updated

Abstract

We propose a method for zero-resource domain adaptation of DNN acoustic models, for use in low-resource situations where the only in-language training data available may be poorly matched to the intended target domain. Our method uses a multi-lingual model in which several DNN layers are shared between languages. This architecture enables domain adaptation transforms learned for one well-resourced language to be applied to an entirely different low- resource language. First, to develop the technique we use English as a well-resourced language and take Spanish to mimic a low-resource language. Experiments in domain adaptation between the conversational telephone speech (CTS) domain and broadcast news (BN) domain demonstrate a 29% relative WER improvement on Spanish BN test data by using only English adaptation data. Second, we demonstrate the effectiveness of the method for low-resource languages with a poor match to the well-resourced language. Even in this scenario, the proposed method achieves relative WER improvements of 18-27% by using solely English data for domain adaptation. Compared to other related approaches based on multi-task and multi-condition training, the proposed method is able to better exploit well-resource language data for improved acoustic modelling of the low-resource target domain.
BibTeX
@inproceedings{abad2020cross,
  author = {Alberto Abad and Peter Bell and Andrea Carmantini and Steve Renals},
  title = {{Cross Lingual Transfer Learning for Zero-Resource Domain Adaptation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=5qfjQrLviZ},
}
Figure