An Application of Recurrent Neural Networks to Discriminative Keyword Spotting

Santiago Fernández, Alex Graves, Jürgen SchmidhuberICANN 2007 · DOI

TL;DR

This paper introduces a novel discriminative keyword spotting system using Bidirectional Long Short-Term Memory (BLSTM) recurrent neural networks trained with Connectionist Temporal Classification (CTC). This approach significantly outperforms traditional HMM-based systems by leveraging long-term dependencies in speech to achieve higher accuracy without requiring explicit threshold tuning.

What Problem Does This Paper Address?

Keyword spotting, the task of detecting specific spoken words in unconstrained speech, is crucial for applications like audio indexing and voice command interfaces. Existing systems often rely on generative Hidden Markov Models (HMMs), which struggle to directly discriminate between keywords and non-keywords. While discriminative approaches using Artificial Neural Networks (ANNs) have emerged, they typically operate on sub-word units (phones) and frame-level posteriors, limiting their ability to capture longer temporal context. This paper addresses the need for a discriminative system that can model entire keywords and utilize long-range acoustic information for more robust detection, especially in challenging, unconstrained speech environments.

Methods

The core of the proposed system is a Bidirectional Long Short-Term Memory (BLSTM) recurrent neural network. BLSTMs are chosen for their ability to process sequential data by considering information from both past and future time steps, which is crucial for understanding speech context. The network is trained using the Connectionist Temporal Classification (CTC) algorithm. CTC is a powerful framework for training sequence-to-sequence models without requiring explicit alignment between input speech frames and output labels. This means the system can learn to detect keywords directly from raw speech, even with variations in pronunciation and timing. The network's output layer is designed to predict the probability of detecting each target keyword or a general "non-keyword" event.

Datasets

The experiments were conducted on the Verbmobil database, specifically a German dialogue dataset focused on scheduling meeting appointments. This database is characterized by spontaneous, unconstrained speech, including long silences and background noise, making it a challenging benchmark. The dataset was divided into training, validation, and test sets, with speakers being independent across these sets. The authors selected twelve keywords related to dates and places for the spotting task.

Key Insights and Ideas

  • Leveraging Long-Term Dependencies: The system's strength lies in its ability to use information from extended time spans to estimate keyword probabilities. This contrasts with frame-level approaches and allows for more robust detection, especially in noisy or variable speech.
  • Discriminative Power: By directly modeling keywords and using a discriminative training approach (CTC), the system can better distinguish between similar-sounding keywords and between keywords and non-keyword speech.
  • Elimination of Manual Thresholding: A significant advantage is that the system does not require a manually set a posteriori threshold to balance true and false positives. The CTC output directly provides probabilities that can be interpreted without this extra tuning step.
  • Flexibility and Scalability: The approach is general and makes few assumptions about the domain, making it adaptable to various keyword spotting tasks. The authors also suggest that the system can be scaled by using multiple smaller networks, potentially one for each keyword or group of similar keywords.

Key Results

The proposed BLSTM-CTC system achieved a keyword spotting accuracy of 84.5% on the Verbmobil test set. This is a substantial improvement over a baseline HMM-based speech recognizer, which achieved only 65% word accuracy on the same data. The system demonstrated high precision, with an average probability of true positives at 0.98 and false positives at 0.80. Notably, the system showed excellent discrimination between similar keywords like "juni" and "juli." Even without using the reference segmentation during evaluation, the system achieved an average accuracy of 86.1%, indicating its robustness.

Limitations and Future Work

The authors acknowledge that for tasks requiring very high precision in the exact location of keywords, adding a constraint to detect keywords within reference segmentation boundaries might be beneficial. However, they argue that this constraint is not necessary for achieving good performance and simplifies data preparation. Future work directions include exploring the use of sub-word units as inputs to the network for improved scalability and the ability to detect unknown keywords. They also propose using multiple specialized networks for different keywords or groups of keywords to further enhance performance and manage complexity.

Summary updated

BibTeX
@inproceedings{fernandez2007application,
  author = {Santiago Fernández and Alex Graves and Jürgen Schmidhuber},
  title = {{An Application of Recurrent Neural Networks to Discriminative Keyword Spotting}},
  booktitle = {ICANN},
  year = {2007},
  doi = {10.1007/978-3-540-74695-9\_23},
}
Figure