A hybrid HMM/DNN approach to keyword spotting of short words

I-Fan Chen, Chin-Hui LeeInterspeech 2013

TL;DR

This paper introduces a novel hybrid HMM/DNN framework for keyword spotting (KWS) specifically designed to improve the detection of short, often function words. By redesigning both the initial keyword hypothesizer and a subsequent verifier with advanced techniques like context-aware HMMs and deep neural networks, they achieve a significant boost in performance, making KWS more robust for real-world applications where short words carry crucial meaning.

What Problem Does This Paper Address?

Keyword spotting (KWS) systems are designed to detect specific keywords within continuous speech, finding applications in areas like speech surveillance and information retrieval. However, a persistent challenge in KWS is the accurate detection of short words. These words, often function words (like "is," "not," "but"), are difficult to detect for several reasons: they have fewer acoustic cues due to their brevity, their pronunciation is heavily influenced by neighboring sounds (co-articulation), and they can be easily confused with partial sounds from other words, leading to high miss rates and numerous false alarms. Existing KWS methods often perform much worse on these short words compared to longer content words, limiting their practical utility.

Methods

The proposed system employs a two-stage approach:

  1. Keyword Hypothesizer (First Stage):
    • Context-Aware Keyword Models: Instead of standard HMMs, they use a context-aware monoword model that concatenates triphone HMMs. This allows for better modeling of pronunciation variations at word boundaries, which are critical for short words.
  2. Redesigned Filler Model: A standard phone-loop filler model can inadvertently absorb short keywords. The authors propose a simplified filler model with fewer states and merged Gaussian mixtures, and later a 9-state filler model. This helps to better distinguish keywords from general speech and reduces false alarms.
  3. MAP Adaptation: Maximum a Posteriori (MAP) adaptation is used to fine-tune both keyword and filler models using training data, improving their accuracy.
  4. Keyword Verifier (Second Stage):
    • MLP/DNN-Based Verification: Putative hits from the hypothesizer are passed to a second-stage verifier, which uses Multi-Layer Perceptrons (MLPs) and Deep Neural Networks (DNNs) to make a final decision.
  5. HMM-State-Alignment (HSA) Feature Transformation: A novel feature transformation technique is introduced. Instead of simply averaging features over equal segments, it uses HMM state alignments to segment the acoustic features of a putative hit. The mean of features within each state is then concatenated, providing a more robust representation.
  6. Knowledge-Based Features: To further enhance verification, they incorporate Government Phonology (GP) features. These features capture articulatory properties of speech sounds and are detected using a recurrent neural network. These GP features are then combined with the standard acoustic features.
  7. Deep Neural Networks (DNNs): For the verifier, they leverage DNNs, which use pre-training techniques (Gaussian-Bernoulli Restricted Boltzmann Machines and Bernoulli Restricted Boltzmann Machines) to initialize model parameters. This helps DNNs perform better than traditional MLPs, especially with multiple hidden layers, leading to more robust detection.

Datasets

  • TIMIT Corpus: The experiments are conducted on the TIMIT corpus, a standard dataset for acoustic-phonetic research. The corpus is divided into training, development, and test sets.
  • Keywords: Nine short keywords (was, with, his, this, from, not, but, every, often) were selected from the TIMIT dictionary based on their occurrence frequency and length (no longer than 4 phones).

Key Insights and Ideas

  • Short words are a distinct challenge: The paper strongly emphasizes that short words require specialized treatment due to their unique acoustic properties and their high impact on overall meaning.
  • Two-stage systems are effective: A hybrid approach, combining a broad hypothesizer with a more discriminative verifier, is crucial for balancing detection rate and false alarm reduction.
  • Context matters at the boundary: For short words, the acoustic information at the word boundaries is paramount. Context-aware HMMs and feature extraction methods that respect these boundaries are key.
  • DNNs offer robustness: The use of DNNs with pre-training significantly improves the verifier's ability to handle variations in short word pronunciations, outperforming single-layer MLPs.
  • Knowledge integration pays off: Incorporating articulatory knowledge (GP features) alongside acoustic features provides valuable complementary information for more accurate verification.

Key Results

The proposed hybrid HMM/DNN system achieves a substantial improvement in the Figure-of-Merit (FOM), a key metric for keyword spotting accuracy.

  • Hypothesizer Improvement: The redesigned keyword hypothesizer (using context-aware models and a 9-state filler) reduced the miss rate from 80.26% (baseline) to 6.08% and increased the FOM from 6.08% to 21.88%.
  • Verifier Improvement:
    • Adding an MLP verifier with HSA transform and MFCC features improved FOM to 27.23%.
  • Incorporating GP features further boosted FOM to 38.08%.
  • Using a DNN verifier with multiple hidden layers and pre-training achieved the best FOM of 42.78%.
  • Comparison: This final FOM of 42.78% is comparable to the FOM for long content words (42.6%) reported in previous research and significantly better than the 18.4% FOM for short keywords in prior work.
  • Runtime: The system maintains a real-time factor (RTF) of approximately 0.02, indicating efficient processing.

Limitations and Future Work

  • Training Data Requirement: The current framework requires selected words to have sufficient occurrences in the training set, which might be difficult for rare content words.
  • Future Work:
    • Testing the approach on spontaneous speech datasets (like Switchboard) to evaluate performance in more realistic, noisy conditions.
  • Exploring a potential third stage for knowledge-based pruning and rescoring using speech attributes (manner, place).
  • Developing alternative strategies for words with insufficient training data.

Summary updated

BibTeX
@inproceedings{chen2013hybrid,
  author = {I-Fan Chen and Chin-Hui Lee},
  title = {{A hybrid HMM/DNN approach to keyword spotting of short words}},
  booktitle = {Interspeech},
  year = {2013},
}
Figure