Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model

Alexander H. Liu, Hung-yi Lee, Lin-Shan Lee

TL;DR

This paper introduces a novel adversarial training framework for end-to-end speech recognition that uses a Criticizing Language Model (CLM) to leverage massive amounts of unpaired text data. By treating the ASR model as a generator and the CLM as a discriminator, the system learns to produce more linguistically plausible transcriptions without requiring extra computation during inference.

What Problem Does This Paper Address?

End-to-end (E2E) Automatic Speech Recognition (ASR) models typically require vast amounts of paired audio-and-transcript data to perform well. However, collecting high-quality paired data is expensive and time-consuming. In contrast, "unpaired" text data (books, news, web text) is nearly infinite.

Existing methods to use this unpaired text have drawbacks:

  1. Language Model (LM) Rescoring: Requires running a separate, heavy LM during testing, which increases latency.
  2. Back-translation: Synthesizes fake speech from text to create "pseudo-paired" data, but the quality of the synthesized audio is often too poor to be truly effective.

The authors seek a way to "bake" linguistic knowledge directly into the ASR model during training so that the final model is both more accurate and remains lightweight for deployment.

Methods

The authors propose an Adversarial Training (AT) approach inspired by Generative Adversarial Networks (GANs). The system consists of two competing components:

  1. The ASR Model (Generator): A standard E2E model (using a VGG+BLSTM encoder and an Attention+LSTM decoder) that converts speech into text sequences.
  2. The Criticizing Language Model (CLM - Discriminator): A CNN-based model designed to distinguish between "real" human text and "fake" transcriptions produced by the ASR model.

The Training Loop

The training happens iteratively in two steps:

  • Step A (Train the Critic): The CLM is shown real text and ASR-generated text. It learns to assign high scores to real text and low scores to ASR outputs.
  • Step B (Train the ASR): The ASR model is trained to minimize its standard recognition loss (comparing its output to ground-truth transcripts) while simultaneously trying to maximize the score it receives from the CLM.

Technical Innovations

  • WGAN-GP: To handle the "discrete vs. continuous" problem (ASR outputs are soft probability distributions, while real text is one-hot encoded), the authors used Wasserstein GAN with Gradient Penalty. This stabilizes training and prevents the CLM from becoming "too good" too quickly, which would stop the ASR model from learning.
  • CNN Architecture for CLM: Instead of an RNN, they used a 1D-CNN for the critic. CNNs are more computationally efficient and excellent at detecting local "unnatural" patterns in text that signal a machine error.

Datasets

The researchers evaluated their approach using the LibriSpeech corpus:

  • Paired Data: 100 hours of clean speech.
  • Unpaired Text: 360 hours of clean text and 500 hours of noisy text.
  • Evaluation: Standard LibriSpeech development and test sets.

Key Insights and Ideas

  • AT as a Regularizer: Surprisingly, adversarial training improved performance even without extra unpaired text. It acts as a sophisticated regularizer, forcing the ASR model to produce outputs that look like natural language rather than just matching character-level patterns.
  • Discarding the Critic: Once training is complete, the CLM is thrown away. The ASR model has implicitly "absorbed" the linguistic knowledge, meaning there is zero overhead during inference compared to a standard model.
  • Grammaticality: Qualitative analysis showed that the AT-trained models were much better at correcting "phonetically similar but grammatically wrong" errors. For example, correcting "NON SENSE" to "nonsense."

Key Results

The adversarial approach showed consistent improvements across various scenarios:

  • Baseline (No extra text): 21.7% Word Error Rate (WER).
  • Adversarial Training (No extra text): 20.1% WER (a 7.4% relative improvement).
  • With 360h Unpaired Text: The AT model reached 19.2% WER, significantly outperforming the Back-translation (BT) method, which only reached 23.6% in their setup.
  • Scaling: When combined with standard RNN-LM fusion and 860 hours of text, the WER dropped to 15.8%, a 27.2% relative improvement over the baseline.

Limitations and Future Work

  • Training Stability: Like most GAN-based approaches, the training is sensitive. The authors noted they had to update the CLM five times less frequently than the ASR model to maintain stability.
  • Vocabulary Constraints: The current work uses subword units. Scaling this to larger vocabularies or different languages might require further tuning of the CLM architecture.
  • Comparison to Large-Scale Pre-training: The paper was published just as large-scale self-supervised learning (like wav2vec) began to emerge. Future work could explore how adversarial training interacts with pre-trained acoustic representations.

Summary updated

Abstract

In this paper we proposed a novel Adversarial Training (AT) approach for end-to-end speech recognition using a Criticizing Language Model (CLM). In this way the CLM and the automatic speech recognition (ASR) model can challenge and learn from each other iteratively to improve the performance. Since the CLM only takes the text as input, huge quantities of unpaired text data can be utilized in this approach within end-to-end training. Moreover, AT can be applied to any end-to-end ASR model using any deep-learning-based language modeling frameworks, and compatible with any existing end-to-end decoding method. Initial results with an example experimental setup demonstrated the proposed approach is able to gain consistent improvements efficiently from auxiliary text data under different scenarios.
BibTeX
@inproceedings{liu2019adversarial,
  author = {Alexander H. Liu and Hung-yi Lee and Lin-Shan Lee},
  title = {{Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=tbN3D7Z6IXL},
}

Loading related content...

Figure