Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Text Data
Lists
Only you see this
Abstract
This paper presents a method to pre-train transformer-based encoder-decoder automatic speech recognition (ASR) models using sufficient target-domain text. During pre-training, we train the transformer decoder as a conditional language model with empty or artifical states, rather than the real encoder states. By this pre-training strategy, the decoder can learn how to generate grammatical text sequence before learning how to generate correct transcriptions. Contrast to other methods which utilize text only data to improve the ASR performance, our method does not change the network architecture of the ASR model or introduce extra component like text-to-speech (TTS) or text-to-encoder (TTE). Experimental results on LibriSpeech corpus show that the proposed method can relatively reduce the word error rate over 10%, using 960 hours transcriptions.
BibTeX
@inproceedings{gao2021pre,
author = {Changfeng Gao and Gaofeng Cheng and Runyan Yang and Han Zhu and Pengyuan Zhang and Yonghong Yan},
title = {{Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Text Data}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2021},
doi = {10.1109/icassp39728.2021.9414080},
}Loading related content...