Semi-Supervised Methods for Improving Keyword Search of Unseen Terms

Scott Novotney, Ivan Bulyko, Richard M. Schwartz, Sanjeev Khudanpur, Owen KimballInterspeech 2012 · DOI

Introduction

In Large Vocabulary Continuous Speech Recognition (LVCSR), systems are typically trained on hundreds of hours of transcribed audio. However, a significant challenge arises when users search for Out-of-Training (OOT) terms—words that exist in the system's dictionary but were rare or absent in the original training data. While the system may know the pronunciation, the Language Model (LM) assigns these terms a very low probability due to a lack of contextual examples. This paper presents a semi-supervised approach to "fill in the holes" of an LM using automatic transcripts of unlabeled audio, achieving substantial performance gains without new human labels.

Proposed Methodology

The researchers explored three primary strategies to increase the visibility of OOT terms:

  1. Unigram Probability Clamping: A "brute force" method that artificially boosts a search term's probability in the LM. While this improves recall, it often leads to "hallucinations" in noisy audio, hurting precision.
  2. Semi-Supervised Language Modeling: The core contribution involves using an existing LVCSR system to transcribe 1,700 hours of unlabeled audio. By extracting $n$-grams from consensus networks (which represent multiple hypotheses) rather than just the "1-best" transcript, the researchers provided the LM with the necessary context for OOT terms.
  3. Directed Transcription (Active Learning): For scenarios with a limited human budget, the authors proposed "directing" transcribers to verify only short snippets where the semi-supervised system predicts a keyword might exist.

Key Insights: LM vs. AM

A critical finding of this research is that the Language Model benefits significantly more from semi-supervised data than the Acoustic Model (AM). LMs suffer from a lack of "parameter sharing"; if a word is missing from training text, the model has zero information about its context. In contrast, the AM can often rely on sub-word phonetic models shared across different words. Narrowing the focus to rare keywords allows for massive gains that are typically obscured when measuring average Word Error Rate (WER) across an entire corpus.

Comparison of AUC improvements from Acoustic Model (AM) and Language Model (LM) updates.

Results and Efficiency

The study was conducted on English Conversational Telephone Speech (CTS), using the Switchboard corpus for training and the Fisher collection for unlabeled evaluation. The baseline system achieved an Area Under the Curve (AUC) of 0.45 for OOT terms. The semi-supervised method raised this to 0.589, capturing 70% of the potential gain that would have been achieved through full manual transcription (AUC 0.648).

Furthermore, the "Directed Transcription" approach proved to be 125 times more cost-effective than undirected manual transcription. As shown in the performance analysis, spending just 270 hours on targeted verification yielded results nearly identical to transcribing the full 34,000-hour equivalent of the Fisher corpus.

AUC vs. Human Effort for different transcription strategies, showing the efficiency of directed methods.

Conclusion and Future Work

This semi-supervised approach effectively addresses the "long tail" of rare keywords in speech recognition. While the LM saw dramatic improvements, the AM remains a potential bottleneck for rare phoneme sequences. Future work may involve applying similar directed techniques to acoustic modeling. Additionally, while demonstrated on English CTS, the robustness of this method in lower-resource languages remains an area for further investigation.

Summary updated

BibTeX
@inproceedings{novotney2012semi,
  author = {Scott Novotney and Ivan Bulyko and Richard M. Schwartz and Sanjeev Khudanpur and Owen Kimball},
  title = {{Semi-Supervised Methods for Improving Keyword Search of Unseen Terms}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-375},
}
Figure