Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings

Matthew Wiesner, Adithya Renduchintala, Shinji Watanabe, Chunxi Liu, Najim Dehak, Sanjeev Khudanpur

Abstract

Attention-based encoder-decoder models for Automatic Speech Recognition (ASR) typically require vast amounts of transcribed data. This paper explores methods to leverage unpaired text data through back-translation-inspired pretraining. By introducing Pseudo-Speech Data Augmentation (PSDA) and pretraining on synthetic "pseudo-speech," the authors demonstrate significant performance gains in low-resource settings, achieving up to a 20% relative improvement in character error rates.

Introduction

Modern end-to-end (E2E) ASR systems are notoriously data-hungry. In low-resource scenarios—where only a few hours of transcribed speech are available—these models often fail to converge or generalize. The primary bottlenecks are twofold: the decoder's inability to learn a robust language model and the attention mechanism's difficulty in learning alignments from limited acoustic-text pairs. While external language models can be integrated via shallow fusion, they do not address the alignment or encoder initialization issues. This work investigates using abundant text data to "warm up" the network before fine-tuning on real audio.

Methodology: MMDA and PSDA

The authors propose using synthetic data derived from text to pretrain the model. Two primary architectures are compared:

  1. Multi-Modal Data Augmentation (MMDA): Text is treated as a secondary input. An augmenting encoder processes phoneme sequences and feeds them directly to the attention mechanism, bypassing the acoustic encoder.
  2. Pseudo-Speech Data Augmentation (PSDA): This variant feeds the output of the augmenting encoder into the primary acoustic encoder. This forces the encoder to process synthetic "pseudo-speech" features, encouraging it to learn useful representations even in the absence of real audio.

The core strategy involves pretraining the entire network (+P) on these synthetic pairs. This initializes the attention and decoder parameters, teaching the model monotonic alignment and target-language syntax early in the training process.

Comparison of MMDA and PSDA architectures.

Experimental Setup

The approach was evaluated on four languages from the Voxforge corpus: Catalan (0.5h), Portuguese (3h), Italian (4h), and French. Additionally, multilingual data from the Hub-4 Spanish corpus was used to test the complementarity of text-based pretraining and traditional multilingual training.

Results and Discussion

The results indicate that pretraining is a critical factor for success in low-resource regimes. Adding pretraining (+P) to either MMDA or PSDA resulted in an average 20% relative improvement in Character Error Rate (CER).

Character Error Rate (CER) results across different tasks and languages.

As shown in the table above, the PSDA + P method significantly outperformed the monolingual baseline. For instance, in the Portuguese (3h) task, the CER dropped from 80.1% to 50.2%. In Italian (4h), the combination of pretraining and multilingual data reached a CER of 17.6%, demonstrating that these techniques are highly complementary.

Key Insights

  • Decoder as a Language Model: Pretraining allows the decoder to learn syntax and vocabulary from text before encountering noisy speech.
  • Alignment Initialization: Synthetic data provides perfect alignment, helping the attention mechanism converge much faster on real data.
  • Efficiency: Text-only pretraining provides approximately 50% of the gains seen from multilingual training without requiring additional audio data.

Conclusion

This paper demonstrates that back-translation-style pretraining on synthetic pseudo-speech is a powerful tool for low-resource ASR. By initializing the model with text-derived knowledge, researchers can significantly reduce the data requirements for building effective speech recognition systems. Future work may involve more sophisticated duration modeling or generative techniques to bridge the gap between synthetic and real acoustic features.

Summary updated

Abstract

We explore training attention-based encoder-decoder ASR in low-resource settings. These models perform poorly when trained on small amounts of transcribed speech, in part because they depend on having sufficient target-side text to train the attention and decoder networks. In this paper we address this shortcoming by pretraining our network parameters using only text-based data and transcribed speech from other languages. We analyze the relative contributions of both sources of data. Across 3 test languages, our text-based approach resulted in a 20% average relative improvement over a text-based augmentation technique without pretraining. Using transcribed speech from nearby languages gives a further 20-30% relative reduction in character error rate.
BibTeX
@inproceedings{wiesner2019pretraining,
  author = {Matthew Wiesner and Adithya Renduchintala and Shinji Watanabe and Chunxi Liu and Najim Dehak and Sanjeev Khudanpur},
  title = {{Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-3254},
}

Loading related content...

Figure