Active Learning for LF-MMI Trained Neural Networks in ASR

Yanhua Long, Hong Ye, Yijie Li, Jiaen LiangInterspeech 2018

TL;DR

This paper introduces a novel active learning (AL) strategy for improving acoustic models in Automatic Speech Recognition (ASR). By combining confidence measures with word matching error rates (WMER) from heterogeneous neural networks, their method efficiently selects the most informative speech utterances for human labeling, significantly reducing the need for manual transcription while achieving substantial improvements in ASR performance.

What Problem Does This Paper Address?

Modern ASR systems heavily rely on deep neural networks (DNNs) for acoustic modeling, leading to significant performance gains. However, as ASR applications diversify, the speech data encountered in real-world scenarios often deviates from the data used for initial model training. This necessitates continuous model updates with fresh speech data. Two primary approaches for this are Semi-Supervised Training (SST) and Active Learning (AL). SST automatically transcribes large amounts of unlabeled data, but these transcriptions can be error-prone and may not capture diverse acoustic properties. AL, on the other hand, guarantees transcription quality through human labeling but faces the challenge of selecting the most informative utterances to minimize costly manual annotation. This paper addresses the core challenge of AL: how to effectively and efficiently select the most valuable data for human labeling to improve ASR models, particularly when dealing with large datasets and diverse acoustic conditions.

Methods

The authors propose an active learning scheme that leverages Heterogeneous Neural Networks (HNNs). Instead of using a single type of neural network for decoding unlabeled speech, they employ two different HNNs (LSTM and TDNN) to generate decoding hypotheses. The core of their method lies in a novel data selection criterion that combines:

  • Word Matching Error Rate (WMER): Calculated by comparing the hypotheses from the two HNNs. This metric helps identify utterances where the models disagree, suggesting potential informativeness.
  • Confidence Measure (CM): Derived from the lattices of one of the HNNs (specifically, the LSTM).

This combined criterion aims to better capture the "disagreement degree" between recognition results compared to methods relying solely on CM. The selected utterances are then manually transcribed and used to retrain the target LF-MMI trained LTDNN acoustic model. A key aspect of their HNN approach is using the entire transcribed base data (TBD) to train the decoding HNNs, which they argue enhances generalization and produces more informative and complementary examples compared to using similar AMs.

Datasets

  • Transcribed Base Data (TBD): Approximately 4000 hours of Mandarin speech data used for training the large-scale ASR baseline and seed models.
  • TBD-add: An additional 600 hours of transcribed data, randomly selected from the 4000 hours, used for small-scale experiments.
  • Untranscribed Fresh Speech Utterances (UTFD): Approximately 30,000 hours of untranscribed speech collected from Unisound's voice search engine. This pool is used for active learning selection. For large-scale experiments, this was increased to 100,000 hours.
  • Test Sets: Three distinct test sets were used for evaluation:
  • Point-of-Interest (POI) speech (approx. 3 hours)
    • General Topic (GTopic) speech (approx. 2.5 hours)
    • Children's speech (approx. 2 hours)

Key Insights and Ideas

  • Heterogeneity is Key: Using multiple, different types of neural networks (HNNs) for decoding unlabeled data is more effective for active learning than using similar models. This diversity helps generate more informative and complementary examples for training.
  • WMER + CM Synergy: Combining WMER (model disagreement) with CM (confidence) provides a more robust criterion for selecting informative utterances than CM alone. This is particularly important because estimating accurate CMs is challenging.
  • Informativeness vs. Difficulty: The paper suggests that utterances that are harder for ASR systems to recognize (higher WMER) are often more informative for training but also require higher transcription costs. The proposed method aims to balance these factors.
  • AL Outperforms SST: For improving acoustic models, especially in scenarios with limited new labeled data or for large-scale tasks, active learning with their proposed method shows greater gains than semi-supervised training.
  • Efficiency Matters: The AL approach significantly reduces the amount of data needed for manual labeling to achieve substantial performance improvements, making it highly practical for real-world ASR systems.

Key Results

  • Relative WER Reduction: The proposed AL scheme achieved a 6-13% relative Word Error Rate (WER) reduction compared to the baseline trained on 4000 hours of data.
  • Data Efficiency: This significant improvement was achieved by selecting and labeling only 1.2K hours of informative utterances from a pool of 100K hours of untranscribed data for large-scale ASR.
  • Comparison with CM-only AL: The CM+WMER criterion led to over 6% relative WER improvements compared to using only confidence measures for data selection on the POI test set.
  • AL vs. SST: In large-scale experiments, adding 7K hours of data via SST yielded only 4-7% performance gains, while the AL approach achieved 6-13% gains with just 1.2K hours of labeled data.
  • Task-Specific Gains: The AL approach showed particular effectiveness in improving acoustic models for challenging evaluation tasks like POI and Child speech, which have more diverse acoustic properties compared to general topics.

Limitations and Future Work

  • Manual Transcription Cost: While significantly reduced, manual transcription is still required for the selected data. The authors suggest that further improvements in HNN models for AL data selection could lead to even greater reductions in transcription cost.
  • Generalization to Other Tasks: The paper focuses on ASR acoustic models. The authors mention that they are investigating the effects of AL on language model training as a future direction.
  • Unified Framework: Initial results combining AL and SST in a unified framework showed further improvements, suggesting this hybrid approach warrants further exploration.

Summary updated

BibTeX
@inproceedings{long2018active,
  author = {Yanhua Long and Hong Ye and Yijie Li and Jiaen Liang},
  title = {{Active Learning for LF-MMI Trained Neural Networks in ASR}},
  booktitle = {Interspeech},
  year = {2018},
}
Figure