Fine-Tuning of Pre-Trained End-to-End Speech Recognition with Generative Adversarial Networks

Md. Akmal Haidar, Mehdi Rezagholizadeh

TL;DR

This paper introduces a novel method for fine-tuning pre-trained end-to-end Automatic Speech Recognition (ASR) models using Generative Adversarial Networks (GANs). By leveraging a pre-trained ASR model as a generator and training a discriminator to distinguish its outputs from ground truth, they achieve improved ASR performance, especially on large datasets where traditional GAN training can be challenging.

What Problem Does This Paper Address?

End-to-end (E2E) ASR systems have become popular due to their simplicity compared to traditional hybrid systems. While Generative Adversarial Networks (GANs) have shown promise in improving ASR robustness, especially in low-resource scenarios, training large E2E ASR models with GANs from scratch is problematic. This is due to the high variance in gradient updates and potential convergence issues that arise from the min-max game, leading to excessively long training times. This paper aims to address this by proposing a more efficient way to leverage GANs for ASR, specifically by fine-tuning an already pre-trained E2E ASR model.

Methods

The core of this paper is a fine-tuning approach for E2E ASR models using GANs. Instead of training a GAN from scratch, they start with an E2E ASR model that has already been pre-trained on a large dataset using standard sequence-to-sequence and CTC losses. This pre-trained ASR model then acts as the generator (G) in the GAN framework. A discriminator (D) is trained to distinguish between the transcriptions generated by the ASR model and the true ground-truth transcriptions.

The key idea is that a pre-trained ASR model, having already learned to approximate the true data distribution, produces "smoother" soft output vectors. This makes it harder for the discriminator to distinguish between real and generated transcriptions, thus forcing the generator (the ASR model) to improve its output to fool the discriminator. The ASR model is then fine-tuned by minimizing its standard losses (CTC and sequence-to-sequence) plus an additional adversarial loss that encourages it to produce outputs that the discriminator classifies as "real." The discriminator and generator are trained alternately.

The discriminator itself is described as an improved version of Wasserstein GAN (WGAN) with gradient penalty, designed to estimate the Earth-Mover distance between real and generated transcriptions. The ASR architecture used is a Transformer-based model, a common choice for state-of-the-art E2E ASR.

Datasets

The experiments are conducted on the LibriSpeech dataset. This dataset is a widely used benchmark for ASR research, containing 960 hours of training data, 10.7 hours of development data, and 10.5 hours of test data, split into "clean" and "other" conditions. No new datasets are introduced.

Key Insights and Ideas

  • Pre-training is crucial for GAN-based ASR fine-tuning: The paper strongly suggests that starting with a pre-trained ASR model significantly improves the GAN fine-tuning process, making it more stable and effective than training a GAN from scratch for ASR on large datasets.
  • Smoother generator outputs aid discriminator training: The hypothesis that pre-trained ASR model outputs are "smoother" and thus harder for the discriminator to classify is key. This difficulty forces the discriminator to become stronger, which in turn provides a more informative adversarial signal for the generator to improve.
  • Adversarial fine-tuning can outperform baselines: The proposed method demonstrates that GAN-based fine-tuning can lead to better Word Error Rate (WER) performance compared to standard pre-trained models and even conventional adversarial training without pre-training.

Key Results

The proposed fine-tuning approach using GANs (Transformer+FTGAN) consistently outperforms baselines on the LibriSpeech dataset for both small and large model settings. Specifically, with LM fusion, the best WER results achieved are:

  • Test-clean: 2.1%
  • Test-other: 4.7%
  • Dev-clean: 1.9%
  • Dev-other: 4.7%

The paper also shows that adversarial training using GANs without pre-training the ASR model (Transformer+GAN) actually leads to a performance drop compared to the baseline, highlighting the importance of the pre-training step.

Limitations and Future Work

Limitations: * The paper notes that adversarial training, even with fine-tuning, can still be computationally intensive, though less so than training from scratch. * The performance drop observed in the "Transformer+GAN" experiment (without pre-training) suggests that the GAN approach is sensitive to the initialization and training strategy, especially on large datasets.

Future Work: * The authors suggest applying VGG-like convolution sub-sampling for further performance improvement. * Exploring other GAN variants or architectural modifications could also be avenues for future research.

Summary updated

Abstract

Adversarial training of end-to-end (E2E) ASR systems using generative adversarial networks (GAN) has recently been explored for low-resource ASR corpora. GANs help to learn the true data representation through a two-player min-max game. However, training an E2E ASR model using a large ASR corpus with a GAN framework has never been explored, because it might take excessively long time due to high-variance gradient updates and face convergence issues. In this paper, we introduce a novel framework for fine-tuning a pre-trained ASR model using the GAN objective where the ASR model acts as a generator and a discriminator tries to distinguish the ASR output from the real data. Since the ASR model is pre-trained, we hypothesize that the ASR model output (soft distribution vectors) helps to get higher scores from the discriminator and makes the task of the discriminator harder within our GAN framework, which in turn improves the performance of the ASR model in the fine-tuning stage. Here, the pre-trained ASR model is fine-tuned adversarially against the discriminator using an additional adversarial loss. Experiments on full LibriSpeech dataset show that our proposed approach outperforms baselines and conventional GAN-based adversarial models.
BibTeX
@inproceedings{haidar2021fine,
  author = {Md. Akmal Haidar and Mehdi Rezagholizadeh},
  title = {{Fine-Tuning of Pre-Trained End-to-End Speech Recognition with Generative Adversarial Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413703},
}
Figure