Adaptive Multi-Corpora Language Model Training for Speech Recognition

Yingyi Ma, Zhe Liu, Xuedong Zhang

Abstract

Neural network language model (NNLM) plays an essential role in automatic speech recognition (ASR) systems, especially in adaptation tasks when text-only data is available. In practice, an NNLM is typically trained on a combination of data sampled from multiple corpora. Thus, the data sampling strategy is important to the adaptation performance. Most existing works focus on designing static sampling strategies. However, each corpus may show varying impacts at different NNLM training stages. In this paper, we introduce a novel adaptive multi-corpora training algorithm that dynamically learns and adjusts the sampling probability of each corpus along the training process. The algorithm is robust to corpora sizes and domain relevance. Compared with static sampling strategy baselines, the proposed approach yields remarkable improvement by achieving up to relative 7% and 9% word error rate (WER) reductions on in-domain and out-of-domain adaptation tasks, respectively.

Introduction

In Automatic Speech Recognition (ASR), Neural Network Language Models (NNLMs) are essential for helping the system understand context and predict the next word. When adapting an ASR system to a specific domain (like medical dictation or social media videos), researchers often mix multiple text datasets because the target-specific data is usually scarce.

The standard industry practice is static sampling: deciding on a fixed ratio for datasets (e.g., 50% social media, 30% news, 20% Wikipedia) and maintaining it throughout the entire training process. However, this is suboptimal. A dataset helpful at the beginning of training for learning basic grammar might be less useful—or even cause overfitting—later when the model needs to learn domain-specific nuances.

Methodology: Adaptive Multi-Corpora Training

The authors propose an Adaptive Multi-Corpora Training framework. Instead of fixing sampling weights at the start, the algorithm treats data selection as an "upper-level" optimization task that runs alongside the "lower-level" model training.

At the start of every training epoch, the algorithm performs a "look-ahead" step:

  1. Individual Fine-tuning: The current model is briefly fine-tuned on each available corpus separately.
  2. Weight Optimization: It calculates how to "blend" these fine-tuned models to achieve the lowest possible error on a small validation set from the target domain.
  3. Dynamic Sampling: The resulting weights become the sampling probabilities for the actual training epoch that follows.

This allows the model to "preview" which datasets are currently most beneficial for improving performance on the target task and prioritize them in the upcoming training cycle.

Key Insights: The Dynamic Learning Curve

A significant finding is the "curriculum" the model discovers. As shown in the figures below, the adaptive algorithm doesn't just pick one dataset and stay there; it adjusts based on the current state of the model.

Validation loss and sampling probability dynamics during training.

In experiments, the algorithm initially focused heavily on in-domain data. In the middle stages, it shifted focus toward out-of-domain data to prevent overfitting and continue learning general patterns. Finally, it settled into a balanced weight. This approach is robust to dataset size; a small but highly relevant dataset will receive a high sampling probability if it contributes significantly to progress.

Experimental Results

The adaptive approach consistently outperformed both uniform weighting and static weights derived from n-gram models across various datasets, including public corpora (Fisher, WSJ, Wikitext-103) and proprietary conversational data.

Performance comparison of different NNLM training methods across multiple corpora.

Key performance metrics include:

  • In-Domain Adaptation: Achieved a relative 7% WER reduction compared to static baselines.
  • Out-of-Domain Adaptation: Achieved a relative 9% WER reduction on conversational tasks and 6% on the Wall Street Journal task.
  • Perplexity (PPL): The model's perplexity dropped significantly, often by 10-20 points compared to standard training methods, indicating much higher confidence in its predictions.

Conclusion and Future Work

While the adaptive method introduces some computational overhead due to the per-epoch fine-tuning "previews," these steps can be parallelized to mitigate training time increases. The method effectively automates curriculum learning without manual tuning or expert intuition. Future work may extend this adaptive sampling logic to end-to-end ASR models, where acoustic and language components are trained jointly.

Summary updated

Abstract

Neural network language model (NNLM) plays an essential role in automatic speech recognition (ASR) systems, especially in adaptation tasks when text-only data is available. In practice, an NNLM is typically trained on a combination of data sampled from multiple corpora. Thus, the data sampling strategy is important to the adaptation performance. Most existing works focus on designing static sampling strategies. However, each corpus may show varying impacts at different NNLM training stages. In this paper, we introduce a novel adaptive multi-corpora training algorithm that dynamically learns and adjusts the sampling probability of each corpus along the training process. The algorithm is robust to corpora sizes and domain relevance. Compared with static sampling strategy baselines, the proposed approach yields remarkable improvement by achieving up to relative 7% and 9% word error rate (WER) reductions on in-domain and out-of-domain adaptation tasks, respectively.
BibTeX
@misc{ma2022adaptive,
  author = {Yingyi Ma and Zhe Liu and Xuedong Zhang},
  title = {{Adaptive Multi-Corpora Language Model Training for Speech Recognition}},
  year = {2022},
  eprint = {2211.05121},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.05121},
}
Figure