Active and Semi-Supervised Learning in ASR: Benefits on the Acoustic and Language Models

Thomas Drugman, Janne Pylkkönen, Reinhard KneserInterspeech 2016

TL;DR

This paper explores how to efficiently train Automatic Speech Recognition (ASR) systems by combining active learning (AL) for selecting data to transcribe and semi-supervised training (SST) for leveraging unlabeled data. They found that AL significantly reduces transcription costs and improves performance, especially when combined with SST, and that sophisticated confidence models don't necessarily offer an advantage over simpler ones.

What Problem Does This Paper Address?

Training high-performance ASR systems requires vast amounts of transcribed speech data. However, transcribing audio is a time-consuming and expensive process. This paper tackles the challenge of minimizing transcription costs while maximizing ASR performance. Specifically, it investigates how to intelligently select which audio segments to have humans transcribe (active learning) and how to effectively use the abundant, untranscribed audio data (semi-supervised training) to improve both the acoustic model (AM) and the language model (LM) of an ASR system. The authors aim to answer whether AL benefits both AM and LM, how AL and SST gains scale with more data, and if different training strategies impact these gains.

Methods

The core of the proposed method relies on confidence-based data selection. This involves using confidence scores to determine which utterances are most informative for training.

  • Confidence Modeling: The paper explores various methods for estimating confidence scores at both the token and utterance levels. This includes using token posteriors from confusion networks, calibrating these scores with models like maximum entropy or neural networks, and using piecewise polynomial or linear regression models to map token posteriors to observed word probabilities. They also investigate training a Multi-Layer Perceptron (MLP) to predict utterance-level confidence.
  • Data Selection:
    • Active Learning (AL): For selecting data to be transcribed by humans (supervised data), the approach focuses on maximizing informativeness by selecting utterances with low confidence scores. They experiment with different confidence filter settings (lower and upper bounds) and also consider filtering out very short utterances.
  • Semi-Supervised Training (SST): For leveraging unlabeled data, the method is inspired by confidence filtering techniques. The goal is to select high-confidence automatically transcribed utterances to avoid introducing too many errors into the training data, which can be detrimental, especially for sequence-discriminative training. Utterance length and frequency filtering are also explored.
  • AM and LM Training:
    • Acoustic Model (AM): A conventional DNN with 4 hidden layers is used. Training involves initial GMM-based alignment, followed by cross-entropy (XE) training and then boosted Maximum Mutual Information (bMMI) sequence-discriminative training. Transfer learning from a pre-trained English DNN is employed.
  • Language Model (LM): A linearly interpolated trigram model is used. The primary component is trained on selected supervised and unsupervised data, while other components are trained on broader Amazon data. Entropy pruning is used to reduce the LM size.

Datasets

  • Training Data: Approximately 600 hours of transcribed German voice search data were used.
  • An initial 50 hours were randomly selected to build the baseline AM and LM.
  • The remaining 550 hours served as the pool for active learning selection and semi-supervised training.
  • Evaluation Data: A held-out dataset of about 8 hours of in-domain German voice search data was used for evaluation.

Key Insights and Ideas

  • Sophisticated Confidence Models Don't Always Win: Surprisingly, more complex confidence models did not yield significant improvements in data selection compared to a simpler polynomial mapping of token posteriors. The authors attribute this to the fact that even if rankings change, the informativeness of the selected utterances remains comparable.
  • AL is Crucial for Both AM and LM: While AL is known to benefit AM training, this paper demonstrates its substantial improvements for LM training as well, especially when combined with SST.
  • SST is Most Valuable Early On: Semi-supervised training provides significant gains when the amount of labeled data is small. However, these gains diminish rapidly as more supervised data becomes available.
  • The Power of Combined AL and SST: The synergy between AL and SST is a key takeaway. This combined approach leads to significant cost reductions and performance improvements, particularly in the initial stages of data collection.
  • Unsupervised Data Can Harm Discriminative Training: For sequence-discriminative training (bMMI), unsupervised data, even with high confidence filtering, can be detrimental due to the risk of propagating errors.

Key Results

  • Cost Reduction: The proposed AL approach allows for a transcription cost reduction of about 70% over random selection for a comparable performance level.
  • Performance Improvement: For a fixed transcription budget (e.g., 100 hours), the proposed approach improves the Word Error Rate (WER) by about 12.5% relative compared to random selection.
  • AM Gains:
    • AL alone reduced WER by about 2% relative over random selection when adding 100 hours of supervised data.
  • Combining AL with SST (200 hours of unsupervised data added to 50 hours of AL) yielded a 4.5% gain on the cross-entropy trained AM.
  • However, unsupervised data was found to be harmful to sequence-discriminative bMMI training.
  • LM Gains: Combining AL and SST allowed for a significant improvement of about 5% in LM performance.
  • Scaling: The benefits of SST are most pronounced at the beginning of data collection, with gains diminishing as more supervised data is acquired. AL's advantage over random selection persists and scales well.

Limitations and Future Work

  • Confidence Model Performance: The paper notes that more sophisticated confidence models did not offer significant advantages for data selection in their experiments. This suggests that simpler models might be sufficient, or that further research is needed to better leverage complex confidence measures.
  • Unsupervised Data for bMMI: The negative impact of unsupervised data on bMMI training highlights a challenge for future work, potentially requiring more robust methods for filtering or incorporating such data.
  • Generalizability: While the study focuses on German voice search, future work could explore the applicability of these methods to other languages and ASR tasks.
  • Further Exploration of Data Selection: The authors suggest that further investigation into the interaction between AL and SST, especially concerning the optimal balance and selection criteria, could be beneficial.

Summary updated

BibTeX
@inproceedings{drugman2016active,
  author = {Thomas Drugman and Janne Pylkkönen and Reinhard Kneser},
  title = {{Active and Semi-Supervised Learning in ASR: Benefits on the Acoustic and Language Models}},
  booktitle = {Interspeech},
  year = {2016},
}
Figure