Small-Footprint Magic Word Detection Method Using Convolutional LSTM Neural Network

Taiki Yamamoto, Ryota Nishimura, Masayuki Misaki, Norihide KitaokaInterspeech 2019

TL;DR

This paper introduces a novel Convolutional LSTM (CLSTM) neural network architecture for "magic word" detection (like "Hey Google" or "Alexa"). Their CLSTM-based models achieve significantly better performance than existing methods while drastically reducing the model size (over 80% parameter reduction), making them ideal for resource-constrained devices.

What Problem Does This Paper Address?

The increasing prevalence of voice-controlled devices (smartphones, smart speakers) necessitates efficient and reliable "magic word" detection. This technology allows devices to wake up and listen for commands without constantly processing all audio, which is crucial for privacy and power conservation. However, existing high-performance detection systems often have large computational footprints, making them unsuitable for embedded systems with limited memory and processing power. The challenge is to achieve high accuracy in detecting these specific keywords from continuous speech while keeping the model small and fast enough to run on such devices.

Methods

The authors propose using Convolutional Long Short-Term Memory (CLSTM) networks for magic word detection. CLSTMs combine the spatial feature extraction capabilities of Convolutional Neural Networks (CNNs) with the temporal sequence modeling of Long Short-Term Memory (LSTM) networks.

  • CNNs are good at capturing local patterns in data, like frequency bands in audio.
  • LSTMs are adept at remembering information over long sequences, crucial for understanding speech context.
  • CLSTMs integrate these by using convolutional operations within the LSTM gates. This allows them to capture spatio-temporal features – patterns that occur across both time and frequency.

They compare their CLSTM-based models against a baseline Convolutional Recurrent Neural Network (CRNN) architecture. Their proposed models explore variations including stacking CLSTMs, or using CLSTMs in conjunction with a Network-in-Network (NIN) layer. They also introduce "sml" (small) variants of their models, specifically optimized to have a very limited number of parameters (around 30,000).

Datasets

The authors constructed a custom dataset for their experiments.

  • Magic Word: The Japanese word "ごえもん" (romanized as "goemon"), chosen for its association with a popular ninja character.
  • Scenarios: Recorded dialogues between two subjects in five different scenarios.
  • Data Processing: Monaural audio streams from lapel microphones were combined into stereo, then quantized and sampled. Speech was converted to log power spectrograms using FFT and Hamming windows.
  • Segmentation: Spectrograms were segmented into frames, with segment sizes of 20 frames (206 ms) and shifts of 10 frames (100 ms).
  • Labeling: Segments containing 70% of the Magic Word were labeled as "Positive" (1), and all others as "Negative" (0).
  • Dataset Size: 187,184 Negative segments (10.711 hours) and 500 Positive segments (103 seconds).
  • Split: The dataset was split into training (90%) and testing (10%) sets.

Key Insights and Ideas

  • CLSTMs Excel at Spatio-Temporal Features: The core idea is that CLSTMs can better capture the complex interplay of frequency and time patterns in speech, which is vital for distinguishing magic words from background noise and other speech.
  • Parameter Reduction is Key for Embedded Systems: The paper strongly emphasizes the need for small models. Their CLSTM architectures, particularly the "sml" variants, achieve dramatic parameter reductions (over 80%) compared to baseline models, making them practical for devices with limited memory.
  • Trade-off Between Performance and Computation: While CLSTMs offer improved accuracy and smaller footprints, they can sometimes require more complex computations (multiplication operations) than simpler LSTMs. The paper highlights the need to balance these factors.
  • Small Models Can Be Highly Effective: The "sml" versions of their CLSTM models demonstrate that significant performance gains can be achieved even with a highly constrained number of parameters.

Key Results

  • Performance Improvement: The proposed CLSTM-based models consistently outperformed the baseline CRNN (lstm_lstm) model in terms of False Rejection Rate (FRr) at a given False Alarm per hour (FAh). For instance, the clstm_nin_2 model showed a 57.5% improvement over the baseline at 25 FAh.
  • Parameter Reduction: The "sml" variants of the proposed models achieved a reduction of over 80% in parameters compared to the baseline model, while still maintaining competitive performance.
  • Real-time Performance on Small Devices: Testing on a Raspberry Pi 3 Model B+ showed that models like clstm_lstm_sml could process speech segments in real-time (under 80 ms per segment), making them suitable for embedded applications.
  • Best Performing Models: clstm_nin_2 and clstm_lstm_sml were identified as particularly strong performers, offering a good balance of accuracy, small footprint, and speed.

Limitations and Future Work

  • Computational Cost: The authors acknowledge that CLSTMs can involve a higher number of multiplication operations compared to simpler models, which can impact processing speed.
  • Future Work:
    • Hyper-parameter Tuning: Further optimization of CLSTM hyper-parameters to reduce computational demands.
  • Arbitrary Magic Word Change: Developing models that allow users to change their magic word dynamically, rather than being fixed to a single word.
  • Faster CLSTM Implementations: Investigating ways to implement CLSTM models more efficiently.

Summary updated

BibTeX
@inproceedings{yamamoto2019small,
  author = {Taiki Yamamoto and Ryota Nishimura and Masayuki Misaki and Norihide Kitaoka},
  title = {{Small-Footprint Magic Word Detection Method Using Convolutional LSTM Neural Network}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure