Hierarchical Neural Network Architecture In Keyword Spotting

Yixiao Qu, Sihao Xue, Zhenyi Ying, Hang Zhou, Jue Sun

Abstract

Keyword Spotting (KWS) provides the start signal of ASR problem, and thus it is essential to ensure a high recall rate. However, its real-time property requires low computation complexity. This contradiction inspires people to find a suitable model which is small enough to perform well in multi environments. To deal with this contradiction, we implement the Hierarchical Neural Network(HNN), which is proved to be effective in many speech recognition problems. HNN outperforms traditional DNN and CNN even though its model size and computation complexity are slightly less. Also, its simple topology structure makes easy to deploy on any device.

Introduction: The Challenge of "Always-On" KWS

Keyword Spotting is the "always-on" technology that listens for wake-words like "Hey Siri" or "Hi Nomi." For a high-quality user experience, KWS must satisfy two opposing requirements: high sensitivity (recall) to catch wake-words in noisy environments, and low computational complexity to minimize battery drain and latency on edge devices. While complex models like LSTMs offer accuracy, they are often too heavy for low-power chips. Standard DNNs and CNNs are faster but struggle to maintain performance across diverse acoustic environments. Researchers at NIO have developed a Hierarchical Neural Network (HNN) architecture that resolves this conflict.

Methodology: The Hierarchical Relay Architecture

The proposed HNN uses a multi-level approach where each level is specialized for different acoustic conditions.

  • Layered Training: The network is trained in three stages using datasets representing different environments: Quiet (Q), Video/Media noise (V), and In-car noise (C).
  • Bottleneck Features: Each level includes a "bottleneck layer"—a narrow layer that compresses information into a low-dimensional representation.
  • Relay Mechanism: As shown in the architecture diagram below, the first level (trained on quiet data) passes its bottleneck features as additional input to the next level. Level 2 takes original audio features plus the bottleneck features from Level 1, and so on. This allows higher levels to benefit from the "clean" representations learned by lower levels.
Figure 1: The HNN architecture showing the relay of bottleneck features (BN) across levels trained on Quiet (Q), Video (V), and Car (C) data.
  • Decoding: A Finite State Transducer (FST) converts the neural network's posterior probabilities into a final "wake" decision based on specific phone units of the keyword.

Experimental Setup

The models were evaluated on a substantial internal dataset of 830,000 utterances (520k positive, 310k negative). The test set comprised 180 hours of data across various noise profiles, specifically targeting Quiet, Video, and In-car environments.

Results and Discussion

The HNN architecture demonstrated significant improvements over traditional models.

  • Superiority over Baselines: The HNN configurations significantly outperformed the baseline 3-layer DNN. As seen in the performance curves, HNN variants consistently achieve a lower "1 - recall rate" for a given false alarm rate (miswake/hour).
  • Bottleneck Strategy: Interestingly, using only one bottleneck feature (passing Level 1 to Level 2 and 3) often performed better than using multiple bottlenecks (allbn), likely due to better model convergence with fewer parameters.
Figure 4: Performance comparison (1-recall vs miswake/hour) between various HNN configurations and the baseline model.
  • Efficiency: HNN models maintained a computational complexity of approximately 0.95M to 1.28M calculations, comparable to the simple baseline, making them ideal for real-time deployment.

Conclusion

The HNN architecture effectively addresses the trade-off between accuracy and complexity in KWS. By specializing levels for different noise profiles and utilizing bottleneck features, it achieves superior robustness. Future work will explore a fully convolutional three-level hierarchy to further enhance feature extraction and accuracy.

Summary updated

Abstract

Keyword Spotting (KWS) provides the start signal of ASR problem, and thus it is essential to ensure a high recall rate. However, its real-time property requires low computation complexity. This contradiction inspires people to find a suitable model which is small enough to perform well in multi environments. To deal with this contradiction, we implement the Hierarchical Neural Network(HNN), which is proved to be effective in many speech recognition problems. HNN outperforms traditional DNN and CNN even though its model size and computation complexity are slightly less. Also, its simple topology structure makes easy to deploy on any device.
BibTeX
@misc{qu2018hierarchical,
  author = {Yixiao Qu and Sihao Xue and Zhenyi Ying and Hang Zhou and Jue Sun},
  title = {{Hierarchical Neural Network Architecture In Keyword Spotting}},
  year = {2018},
  eprint = {1811.02320},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.02320},
}
Figure