An Investigation of Mixup Training Strategies for Acoustic Models in ASR

Ivan Medennikov, Yuri Khokhlov, Aleksei Romanenko, Dmitry Popov, Natalia Tomashenko, Ivan Sorokin, Alexander Zatvornitskiy2018 · DOI

TL;DR

This paper explores the effectiveness of "mixup," a data augmentation technique, for training acoustic models in Automatic Speech Recognition (ASR). They demonstrate that mixup consistently improves ASR performance across various datasets and model architectures, particularly when test data differs from training data, offering a low-cost way to boost model robustness.

What Problem Does This Paper Address?

Modern Automatic Speech Recognition (ASR) systems heavily rely on neural networks. A common challenge in training these networks is overfitting, where the model memorizes the training data instead of learning generalizable patterns. This leads to poor performance on unseen data, especially when there are mismatches between training and testing conditions (e.g., different acoustic environments, speakers, or accents). While techniques like data augmentation (e.g., speed perturbation) and regularization (e.g., dropout) exist, this paper investigates a newer technique, "mixup," as a potential solution to enhance generalization and robustness in ASR acoustic models.

Methods

The core of this paper is the application and evaluation of "mixup" for acoustic model training. Mixup, introduced in [4], works by creating synthetic training examples through linear interpolation of existing input features and their corresponding labels. The authors explore various strategies for applying mixup to ASR, considering two main training paradigms:

  • Cross-Entropy (CE) Training: They experimented with several mixup schemes, including:
  • Global Scheme: Linearly interpolating features and labels for entire sequences.
  • Local Scheme: Mixing frames within a short window.
  • Class Scheme: Mixing frames with randomly chosen frames of the same label.
  • Shift Scheme: Mixing a sequence with a time-shifted version of itself.
  • Lattice-Free Maximum Mutual Information (LF-MMI) Training: This is a more advanced sequence-discriminative training method. For LF-MMI, they propose a "gradient mixup" approach, which involves mixing intermediate results during the forward-backward pass. They also explore simpler methods of mixing input sequences and their corresponding numerator graphs, with different scaling strategies for the weights.

The authors also compare mixup against established techniques like speed perturbation (data augmentation) and dropout (regularization).

Datasets

The paper uses three popular ASR benchmarks to evaluate their methods:

  • Switchboard-1 (SWBD): Conversational telephone speech, used for both CE and LF-MMI experiments. This dataset is further divided into SW (similar to training) and CH (CallHome English, significantly different from training) for evaluating robustness.
  • TED-LIUM: Lectures, used for LF-MMI experiments.
  • AMI: Meeting corpus, used for LF-MMI experiments, specifically for Individual Headset Microphone (IHM) and Single Distant Microphone (SDM) tasks.

Key Insights and Ideas

  • Mixup is a versatile tool: It shows consistent improvements across different acoustic modeling approaches (CE and LF-MMI) and diverse datasets.
  • Robustness to mismatch: Mixup is particularly effective when test data differs from training data, suggesting it helps models generalize better to unseen conditions.
  • Complementary to other techniques: Mixup can be used alongside existing data augmentation (speed perturbation) and regularization (dropout) methods, potentially leading to further gains.
  • Simplicity and low cost: Mixup is relatively easy to implement and adds minimal overhead to the training process, making it an attractive option.
  • Scheme matters: For CE training, the "global scheme" of mixup was most effective, while other schemes showed less benefit or even degraded performance. For LF-MMI, the differences between mixup schemes were less pronounced.

Key Results

  • Relative WER reduction: The paper reports consistent improvements in Word Error Rate (WER), with up to a 13% relative reduction across various ASR tasks.
  • CE Training: On Switchboard, the "global" mixup scheme achieved a WER of 15.0% (all-condition), compared to 15.2% for the baseline.
  • LF-MMI Training: On Switchboard, mixup achieved a WER of 13.8% (all-condition) compared to the baseline of 14.9%. On TED-LIUM, mixup achieved a WER of 10.1% (test2) compared to the baseline of 11.1%.
  • Comparison with Speed Perturbation: Mixup performed comparably to speed perturbation for CE models but significantly outperformed it for LF-MMI models on Switchboard.
  • Comparison with Dropout: Mixup generally outperformed dropout. Combining them sometimes led to degradation, but dropout with a low probability showed some benefit when combined with mixup.
  • Robustness: Mixup significantly reduced WER on the CallHome (CH) portion of the Switchboard dataset, highlighting its benefit for mismatched conditions.

Limitations and Future Work

  • Gradient Mixup Complexity: The proposed "gradient mixup" for LF-MMI training is computationally expensive. The authors plan to explore simplified versions of this approach.
  • Further Mixup Schemes: Future work will involve deeper investigation into various mixup schemes, including different mixture weight distributions and scheduling strategies.
  • Language Models: The authors express interest in applying mixup to neural network-based language models for ASR.
  • Specific Mixup Schemes for CE: While the "global" scheme was effective for CE, further exploration of other schemes or hybrid approaches might yield additional benefits.

Summary updated

Abstract

International audience
BibTeX
@misc{medennikov2018investigation,
  author = {Ivan Medennikov and Yuri Khokhlov and Aleksei Romanenko and Dmitry Popov and Natalia Tomashenko and Ivan Sorokin and Alexander Zatvornitskiy},
  title = {{An Investigation of Mixup Training Strategies for Acoustic Models in ASR}},
  year = {2018},
  doi = {10.21437/interspeech.2018-2191},
}
Figure