Active learning based data selection for limited resource STT and KWS

Thiago Fraga-Silva, Jean-Luc Gauvain, Lori Lamel, Antoine Laurent, Viet-Bac Le, Abdel MessaoudiInterspeech 2015

TL;DR

This paper tackles the challenge of building speech recognition (STT) and keyword spotting (KWS) systems for low-resource languages, where labeled data is scarce. They propose an active learning approach to intelligently select which unlabeled audio data to transcribe manually, significantly outperforming baseline methods and improving system performance.

What Problem Does This Paper Address?

Developing robust speech technologies for languages with limited digital presence (low-resource languages) is a significant challenge. Traditional methods require vast amounts of transcribed audio, which is expensive and time-consuming to acquire for these languages. The IARPA-Babel program aims to accelerate the development of STT and KWS systems for such languages. A key bottleneck is the manual transcription of audio data. This paper addresses the problem of efficiently selecting the most informative unlabeled audio segments from a large pool to be transcribed, thereby maximizing the performance gain for a given transcription budget.

Methods

The core of this work is an Active Learning (AL) strategy for data selection. Instead of randomly picking data for transcription, AL aims to select data that the current model is most uncertain about or that would provide the most new information.

Here's a breakdown of their approach:

  • Bootstrap System: An initial STT system is trained on a very small amount of transcribed data (1 hour).
  • Data Pool: A larger pool of untranscribed audio data (29 hours) is available.
  • Selection Criteria: The bootstrap system is used to generate hypotheses for the untranscribed data. Various criteria are then applied to this decoded data to select segments for manual transcription. These criteria include:
  • HMM-state entropy: Measures the uncertainty in the acoustic model's state distribution for a given segment. Higher entropy suggests more novel acoustic information.
  • Vocabulary size: The number of distinct words in the decoded hypotheses.
  • Log-likelihood: The probability of the data under the current model.
  • Letter density: The ratio of letters to speech duration.
  • And others like signal duration, confidence scores, etc.
  • Iterative Refinement (Simulated): While the paper focuses on a single iteration for selection, the AL paradigm implies that the newly transcribed data would be used to retrain the system, which would then be used for further selection. For this study, they simulated this by using an internal transcription recovery algorithm on already transcribed data.
  • Baseline Comparison: The AL-selected data (totaling 3 hours: 1 hour initial + 2 hours selected) is compared against a Very Limited Language Pack (VLLP) baseline, which uses a pre-defined 3-hour set of data.

Datasets

The experiments were conducted on data from the IARPA-Babel program, specifically focusing on the OP2 phase which includes languages like Cebuano, Kazakh, Kurdish, Lithuanian, Telugu, and Tok-Pisin.

  • Training Data:
    • Initial bootstrap set: 1 hour of transcribed audio per language.
  • AL pool: 29 hours of untranscribed audio per language.
  • VLLP baseline: A pre-defined 3-hour set of transcribed audio per language.
  • Total transcribed data for AL systems: 3 hours (1h initial + 2h selected).
  • Textual Corpus: A large web-based textual corpus (e.g., Wikipedia, subtitles) was also available for language model training, varying in size from 5.7M to 49M words per language. For Lithuanian, it was about 26M words.
  • Evaluation Data:
    • Tuning set: 3 hours of speech for system parameter optimization.
  • Development set (DEV): 10 hours of speech for evaluating STT and KWS performance.

Key Insights and Ideas

  • Active Learning is Key for Low-Resource Languages: The fundamental idea that intelligently selecting data for transcription can yield better results than random selection or fixed datasets is validated. This is crucial for resource-constrained scenarios.
  • HMM-state Entropy as a Strong Predictor: The paper highlights HMM-state entropy as a particularly effective criterion for data selection. It suggests that segments with higher acoustic model uncertainty are more valuable for improving the model.
  • Vocabulary Diversity Matters: Selecting data that introduces a wider variety of words into the hypotheses also strongly correlates with improved performance. This implies that AL should aim to cover more of the language's lexical space.
  • Web Data Boosts Language Models: Incorporating large web-based textual corpora significantly improves language model performance, leading to substantial gains in STT accuracy, even when the acoustic model training data is limited.
  • Correlation vs. Direct Optimization: While metrics like WER and ATWV are the ultimate goals, they cannot be directly used for selection because true transcriptions are unknown. The paper explores proxies (like entropy and vocabulary size) that are correlated with these metrics.
  • Confidence Scores are Less Reliable: Interestingly, confidence scores from the initial bootstrap system showed little correlation with WER, possibly due to the low accuracy of the initial model.

Key Results

  • STT Performance: AL-based systems, particularly those using HMM-state entropy or letter density for selection, outperformed the VLLP baseline by 1-1.7% absolute in terms of Word Error Rate (WER) on the Lithuanian development set.
  • KWS Performance: AL-based systems also improved over the VLLP baseline in Actual Term-Weighted Value (ATWV) by approximately 3% absolute.
  • Web Data Impact: Using web-based language models (e.g., 100k-word list) led to a significant WER reduction of 4.8% absolute on the Lithuanian TUN data compared to using only manual transcriptions.
  • Selection Criteria Performance: The HMM-state entropy and letter density criteria were found to be the most effective for data selection, recovering the largest vocabularies and leading to the best WER.

Limitations and Future Work

  • Single Iteration: The experiments were conducted with a single AL iteration. The authors suggest that further iterations could yield even greater improvements.
  • Simulated Transcription Recovery: The selection was based on recovered transcripts from an already transcribed pool. In a real-world scenario, the selection would be made from truly untranscribed data, and the subsequent transcription would be manual.
  • Combining Criteria: While individual criteria showed promise, the authors note that combining multiple selection criteria could potentially lead to further performance gains. This is a promising direction for future work.
  • Real-World Transcription Constraints: The authors mention that in real conditions, adding constraints on minimum and maximum segment duration might be beneficial for manual transcription efforts.
  • Generalizability: While similar results were observed for other IARPA-Babel languages, further testing and analysis across a wider range of languages would be valuable.

Summary updated

BibTeX
@inproceedings{fragasilva2015active,
  author = {Thiago Fraga-Silva and Jean-Luc Gauvain and Lori Lamel and Antoine Laurent and Viet-Bac Le and Abdel Messaoudi},
  title = {{Active learning based data selection for limited resource STT and KWS}},
  booktitle = {Interspeech},
  year = {2015},
}
Figure