Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model

Alexander H. Liu, Hung-yi Lee, Lin-Shan Lee

Abstract

In this paper we proposed a novel Adversarial Training (AT) approach for end-to-end speech recognition using a Criticizing Language Model (CLM). In this way the CLM and the automatic speech recognition (ASR) model can challenge and learn from each other iteratively to improve the performance. Since the CLM only takes the text as input, huge quantities of unpaired text data can be utilized in this approach within end-to-end training. Moreover, AT can be applied to any end-to-end ASR model using any deep-learning-based language modeling frameworks, and compatible with any existing end-to-end decoding method. Initial results with an example experimental setup demonstrated the proposed approach is able to gain consistent improvements efficiently from auxiliary text data under different scenarios.

Introduction

End-to-end (E2E) automatic speech recognition (ASR) has revolutionized the field but remains constrained by a heavy reliance on large-scale paired audio-transcription datasets. While paired data is expensive to produce, unpaired text data is abundant. Traditional methods to leverage this text, such as shallow fusion or back-translation, often introduce computational overhead during inference or suffer from the low quality of synthetic speech. This work introduces an adversarial training framework that integrates linguistic knowledge directly into the ASR model during training, maintaining a lean model for deployment while improving accuracy.

Methods

The proposed framework utilizes an Adversarial Training (AT) loop involving two primary components: an ASR Model (Generator) and a Criticizing Language Model (CLM - Discriminator). The ASR model is a hybrid CTC/Attention architecture that aims to minimize standard recognition loss while maximizing a "quality score" assigned by the CLM. The CLM is a 1D-Convolutional Neural Network (CNN) designed to distinguish between real human text and the "soft" probability distributions produced by the ASR model.

The iterative learning process between the CLM and the ASR model.

To handle the discrete nature of text, the authors employ the Wasserstein GAN with Gradient Penalty (WGAN-GP). By operating on soft distributions rather than hard word choices, the framework allows for stable backpropagation, enabling the ASR model to learn complex linguistic patterns from unpaired text. This process forces the ASR encoder and decoder to internalize grammar and vocabulary during the initial learning phase, acting as a powerful regularizer.

Results

The approach was evaluated on the LibriSpeech corpus using 100 hours of paired data and up to 860 hours of unpaired text. Experimental results demonstrate that the CLM provides substantial gains when auxiliary text is included, outperforming standard baselines.

Performance comparison (CER/WER) across different data configurations and methods.

As shown in the results table, the baseline WER of 21.7% on the test set was reduced to 18.7% through adversarial training alone (+AT). When combined with traditional shallow fusion (+Both), the model achieved a 15.8% WER, representing a 27.2% relative improvement over the baseline. Notably, the AT approach consistently outperformed the popular back-translation (BT) method under identical data conditions. Qualitative analysis further showed that the AT model was significantly better at correcting phonetic errors into grammatically correct words (e.g., correctly identifying "Alice" where the baseline produced phonetic gibberish).

Conclusion

This research demonstrates that adversarial training with a criticizing language model is a powerful and efficient way to utilize unpaired text data in E2E ASR. By internalizing linguistic constraints during training, the final model achieves higher accuracy without increasing inference-time complexity. Future work will explore the scalability of this approach to larger paired datasets and its applicability to different subword or character-level modeling units.

Summary updated

Abstract

In this paper we proposed a novel Adversarial Training (AT) approach for end-to-end speech recognition using a Criticizing Language Model (CLM). In this way the CLM and the automatic speech recognition (ASR) model can challenge and learn from each other iteratively to improve the performance. Since the CLM only takes the text as input, huge quantities of unpaired text data can be utilized in this approach within end-to-end training. Moreover, AT can be applied to any end-to-end ASR model using any deep-learning-based language modeling frameworks, and compatible with any existing end-to-end decoding method. Initial results with an example experimental setup demonstrated the proposed approach is able to gain consistent improvements efficiently from auxiliary text data under different scenarios.
BibTeX
@inproceedings{liu2019adversarial,
  author = {Alexander H. Liu and Hung-yi Lee and Lin-Shan Lee},
  title = {{Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=tbN3D7Z6IXL},
}
Figure