Phrase-Based Query Degradation Modeling for Vocabulary-Independent Ranked Utterance Retrieval

J. Scott Olsson, Douglas W. OardNAACL 2009

TL;DR

This paper introduces a novel approach to finding spoken words in audio archives, even when those words aren't in the standard speech recognition dictionary (Out-Of-Vocabulary or OOV words). By modeling potential "degradations" or misrecognitions of query words using phrase-based machine translation, they significantly improve retrieval accuracy, especially for rare and informative OOV terms.

What Problem Does This Paper Address?

Finding specific spoken words within large audio collections is a crucial task for many applications, from historical archives to intelligence analysis. A major challenge arises when the words users search for are "Out-Of-Vocabulary" (OOV) – meaning they aren't part of the speech recognition system's built-in dictionary. This is particularly common for rare words, proper nouns, or specialized terminology. Traditional speech recognition systems struggle with OOV words, leading to poor retrieval performance. Existing methods for OOV word retrieval often rely on phoneme-level recognition, which itself can be error-prone. This paper aims to bridge this gap by developing a more robust method for handling recognition errors and finding OOV words.

Methods

The core innovation is Query Degradation Modeling. Instead of just searching for the exact phoneme sequence of a query word, the system hypothesizes many plausible "degradations" – alternative phoneme sequences that might result from speech recognition errors.

  • Baseline: The paper first establishes a baseline using a generative model that estimates term frequency within speech recognition lattices (a representation of multiple recognition hypotheses). They compare different indexing units: phonemes, phoneme multigrams (sequences of phonemes), and words.
  • Query Degradation: The key new approach involves modeling the probability of a degraded phoneme sequence given the original query phoneme sequence, denoted as $P(H|Q)$.
  • CMQD (Confusion Matrix based Query Degradation): A simpler baseline degradation model using phoneme confusion matrices derived from alignments.
  • PBQD (Phrase-Based Query Degradation): The main contribution, leveraging phrase-based statistical machine translation (SMT). This treats the problem as translating a "source" reference phoneme sequence into a "target" hypothesized phoneme sequence. This allows for more flexible substitutions, insertions, and deletions, and can incorporate linguistic knowledge through "factors" (e.g., broad phoneme classes like vowels/consonants).
  • Indexing: The system indexes speech using expected counts of phoneme sequences from recognition lattices. They explore indexing with phonemes, phoneme multigrams, and words.

Datasets

  • Training Data: Approximately 200 hours of transcribed audio from the MALACH collection (oral history interviews of Holocaust survivors and witnesses). A held-out set of 22,810 manually transcribed utterances was used for training degradation models.
  • Evaluation Data: Seventeen fully transcribed MALACH interviews were used for evaluation.
  • Query Set: The query set consisted of all single words occurring in the evaluation interviews that were OOV with respect to the word-based LVCSR dictionary. This resulted in 261 query terms.

Key Insights and Ideas

  • OOV words are informative: Rare words, often the most informative, are also the most likely to be OOV, making robust OOV retrieval critical.
  • Recognition errors are inevitable: Phoneme recognition is inherently noisy, and even word-based recognition can produce errors that lead to OOV issues.
  • Phrase-based SMT is powerful for modeling errors: Applying SMT techniques to hypothesize phoneme sequence degradations allows for more sophisticated modeling of recognition errors than simple confusion matrices. It can capture multi-phoneme substitutions and leverage language model statistics effectively.
  • Longer recognition units improve baseline: Using phoneme multigrams or words for indexing significantly improves baseline retrieval performance compared to using individual phonemes, likely due to better phoneme recognition.
  • Degradation modeling complements longer units: Query degradation modeling provides further significant improvements on top of using longer recognition units, demonstrating its value in handling residual recognition errors.

Key Results

  • Improved MAP: The phrase-based query degradation approach (PBQD-Fac) consistently improved Mean Average Precision (MAP) across all indexing methods (phonemes, multigrams, words) compared to the baseline generative model.
  • Best Performance with Words + PBQD-Fac: The best overall MAP was achieved using word-based indexing combined with the PBQD-Fac degradation model, reaching 0.1787.
  • Multigram Advantage: Multigram indexing showed promise, achieving competitive MAP with the word-based system and offering smoother phoneme language models.
  • CMQD vs. PBQD: While CMQD showed some improvement, PBQD (especially PBQD-Fac) demonstrated a more consistent and significant increase in MAP, particularly with a larger number of degradations considered.
  • PER: The Phoneme Error Rate (PER) was significantly lower for multigram (32.1%) and word (20.5%) systems compared to phoneme-only (64.4%), highlighting the benefit of longer units.

Limitations and Future Work

  • CMQD vs. PBQD-Fac for specific queries: While PBQD-Fac generally outperformed CMQD, Table 2 shows an example where CMQD achieved higher precision for a specific query ("Mengele"). This suggests that the phrase-based language model might sometimes exert too much influence, leading to false alarms.
  • False Alarm Model: The authors suggest incorporating a false alarm model to down-weight occurrences that look suspicious but are not true query words.
  • Discriminative Framework: Training the degradation model in a discriminative framework to directly penalize false alarms is proposed as a future direction.
  • In-vocabulary words: The paper focuses on OOV words, but future work could incorporate evidence from in-vocabulary query words for downstream tasks like ad hoc speech retrieval.
  • Tuning degradation count: The optimal number of degradations to consider might be task-specific.

Summary updated

BibTeX
@inproceedings{olsson2009phrase,
  author = {J. Scott Olsson and Douglas W. Oard},
  title = {{Phrase-Based Query Degradation Modeling for Vocabulary-Independent Ranked Utterance Retrieval}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure