White listing and score normalization for keyword spotting of noisy speech

Bing Zhang, Richard Schwartz, Stavros Tsakalidis, Long Nguyen, Spyros MatsoukasInterspeech 2012

TL;DR

This paper introduces two novel techniques to improve keyword spotting (KWS) in noisy speech: "white listing" to ensure keywords are never pruned from recognition results, and "pFA score normalization" to make keyword scores consistent across different words. These methods significantly reduce miss rates for keywords in challenging acoustic conditions, outperforming existing sub-word approaches.

What Problem Does This Paper Address?

Keyword spotting (KWS) systems aim to detect specific words or phrases within audio. Traditional KWS systems often rely on large-vocabulary continuous speech recognition (LVCSR) systems. However, two major challenges arise:

  1. Missed Keywords: LVCSR systems can prune keywords from their search or output lattices, especially in noisy or degraded speech, leading to high miss rates. Even with techniques to increase recall (like wider search beams), computation costs can become prohibitive.
  2. Inconsistent Scores: When searching for multiple keywords, their confidence scores from the LVCSR system are not directly comparable. This makes it difficult to rank keywords or set a universal detection threshold, as a "good" score for one keyword might be poor for another. This is exacerbated by the fact that keywords might appear very rarely in training data, making it hard to calibrate their scores.

The paper specifically addresses these issues in the context of speech degraded by transmission through various noisy radio channels, a common scenario in real-world applications.

Methods

The paper proposes two main innovations:

  1. White Listing: This technique involves explicitly informing the speech recognizer about the set of target keywords. During the decoding process, the pruning threshold for these "white-listed" words is significantly relaxed (a wider beam). This ensures that even if a keyword is spoken with low confidence or in a noisy segment, it is highly unlikely to be pruned from the search lattice. While this increases computation for the keywords themselves, the overall system's computational overhead remains manageable because the number of keywords is typically small.
  2. pFA Score Normalization: This method addresses the inconsistent scoring problem. Instead of directly using the raw scores from the recognizer, the authors propose mapping these scores to the probability of false alarm (pFA) for each keyword. This mapping is learned by running the KWS system on a large corpus of transcribed speech (even if it doesn't contain the target keywords) and observing how many false alarms occur at different score levels. By sorting these false alarms by score, an empirical pFA can be calculated for each score bin. This allows for a consistent score across all keywords, as a given pFA value will correspond to the same performance level regardless of the keyword. The paper uses 1-pFA as the final score, so higher scores are better.

Datasets

  • Training Data:
    • Levantine Fisher Conversational Telephone Speech (LFCTS)
    • Retransmissions of LFCTS subsets over 8 degraded radio channels.
  • Acoustic training corpus for experiments (R1 and R2 releases) totaling 196-290 hours.
  • Language model training corpus: acoustic training transcripts (504K words) plus Levantine STT and web text (1M words).
  • Test Data:
    • Dev04 test set from the DARPA EARS program.
  • Dev04r: a retransmitted version of Dev04.
  • Dev-1: official development set for phase-1 of the DARPA RATS project, consisting of 1.1 hours of noisy speech with 219 unique keywords.
  • "Tune" and "Validation" sets created from release R2 for fair evaluation of score normalization.

Key Insights and Ideas

  • White listing is surprisingly effective: By simply preventing keywords from being pruned, the system can achieve much lower miss rates than complex sub-word models, even with severely degraded speech. This suggests that leveraging whole-word LVCSR models, when their outputs are preserved, can be more robust than relying on inherently less accurate sub-word units.
  • pFA normalization creates a universal scoring scale: The ability to estimate a reliable pFA for any keyword, even those absent from training data, is crucial. This allows for a unified performance metric and thresholding across all keywords, simplifying system integration and evaluation.
  • The problem of phrase vs. word competition is alleviated: The pFA normalization implicitly handles the issue where individual words within a keyword phrase might get high scores, potentially lowering the score for the entire phrase. The normalization adjusts for this by considering the empirical relationship between scores and false alarms.
  • Fair evaluation is critical: The paper highlights the importance of evaluating score normalization on separate "tune" and "validation" sets to avoid "cheating" by training the normalizer on the test data itself.

Key Results

  • White Listing: The word-based KWS system with white listing achieved a recall of 98% on the Dev-1 test set, significantly outperforming individual sub-word systems and the baseline system combination. Table 2 shows a substantial reduction in miss rate (pMiss) for the white-listed system compared to the baseline, particularly at low false alarm rates. For example, at 0.2% pFA, pMiss dropped from 45.4% (baseline word) to 33.7% (white list word).
  • Score Normalization: pFA-based score normalization led to very large improvements. Table 3 shows that without normalization, pMiss was 30.0% at 0.2% pFA, which dropped to 25.2% with pFA normalization. Table 4 demonstrates that even under fair evaluation conditions (using separate tune/validation sets), the pFA normalization still provided significant gains, reducing pMiss from 37.0% to 33.2% at 0.2% pFA.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while white listing prevents pruning, the individual words within a keyword phrase can still compete with the whole phrase, affecting posterior probabilities. While pFA normalization alleviates this, it doesn't entirely remove the problem.
  • The white listing method increases computation for the keywords themselves, though the overall system overhead is managed.
  • Future Work:
    • The paper suggests that the pFA score normalization method can be used for combining multiple KWS systems by applying it to the output of each system and then interpolating the normalized scores. This is mentioned as an alternate method to traditional weighted combinations.

Summary updated

BibTeX
@inproceedings{zhang2012white,
  author = {Bing Zhang and Richard Schwartz and Stavros Tsakalidis and Long Nguyen and Spyros Matsoukas},
  title = {{White listing and score normalization for keyword spotting of noisy speech}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure