Stochastic Adaptive Neural Architecture Search for Keyword Spotting

Tom Véniat, Olivier Schwander, Ludovic Denoyer

Abstract

Keyword spotting (KWS) is a critical component for always-on voice interfaces, typically requiring a neural network to process continuous audio streams. However, standard static architectures often waste significant energy processing silence or background noise. We present Stochastic Adaptive Neural Architecture Search (SANAS), a framework that dynamically adjusts a neural network's architecture at inference time. By using a recurrent policy to sample sub-networks based on the audio context, SANAS allocates high computational resources to complex speech segments and minimal resources to background noise. Our experiments on the Speech Commands dataset demonstrate that SANAS achieves state-of-the-art accuracy while reducing computational costs by up to 70% compared to static baselines.

Introduction

Keyword spotting (KWS) systems, such as those used for "Hey Siri" or "OK Google," must operate continuously under strict power constraints. Traditionally, this is achieved by sliding a fixed-size neural network over an audio stream. This approach is inherently inefficient because the computational cost remains constant regardless of whether the input is a complex keyword, simple background noise, or silence.

Existing Neural Architecture Search (NAS) methods focus on finding a single optimal static architecture. In contrast, we propose a dynamic approach where the model's complexity adapts to the difficulty of the input signal. This "on-the-fly" adaptation allows for significant energy savings on mobile and IoT devices without sacrificing recognition performance.

The SANAS Framework

SANAS treats architecture selection as a dynamic, recurrent process. The framework is built upon a "Super-Network"—a large graph containing numerous possible sub-graphs of varying depths and widths. In our implementation, we utilize the popular cnn-trad-fpool3 architecture as a base and augment it with multiple shortcut connections to create a rich search space.

At each timestep $t$, a Recurrent Neural Network (RNN) maintains a latent state $z_t$ representing the audio context. This RNN generates a distribution over possible architectures, from which a specific sub-graph is sampled. To train this non-differentiable sampling process, we employ Monte-Carlo gradient estimation (similar to the REINFORCE algorithm) with variance reduction. The model is optimized using a budgeted loss function that balances prediction accuracy against the average computational cost (FLOPs).

Context-Aware Resource Allocation

A key insight of SANAS is its ability to learn word-specific and noise-specific computational budgets. During the training process, the model naturally discovers that background noise requires far less "thinking time" than actual speech.

Evolution of computational cost (FLOPs) per keyword and background noise during training. The model learns to allocate significantly fewer resources to background noise over time.

As shown in the figure above, the FLOP count for background noise frames (the orange line) drops dramatically as training progresses, eventually settling at a fraction of the cost required for keywords. This demonstrates that the model successfully learns to "wake up" only when the input signal contains potentially relevant information.

Experimental Results

We evaluated SANAS on the Speech Commands dataset, simulating a real-world streaming environment by mixing background noise with randomly located words. The results indicate that dynamic architectures consistently outperform static ones in terms of the accuracy-cost trade-off.

Accuracy vs. computational cost (FLOPs per frame). SANAS (green circles) achieves a superior trade-off compared to static baselines (crosses).

As illustrated in the trade-off plot, SANAS configurations (green circles) occupy a superior Pareto front compared to the static cnn-trad-fpool3 baselines. For instance, one SANAS variant achieved an 86.5% match rate using only 37.7M FLOPs per frame, whereas the standard static baseline required 124.6M FLOPs to achieve a lower 81.7% match rate. In its most aggressive power-saving mode, SANAS can operate at just 519K FLOPs—a 240x reduction from the baseline—while still maintaining a functional 61.2% match rate for low-power wake-up scenarios.

Conclusion

SANAS represents a significant step toward efficient, always-on keyword spotting. By moving from static to adaptive architectures, we can achieve higher accuracy with a fraction of the computational overhead. Future work will focus on translating these theoretical FLOP savings into actual battery life improvements on specialized edge hardware and exploring even larger search spaces for more complex audio tasks.

Summary updated

Abstract

The problem of keyword spotting i.e. identifying keywords in a real-time audio stream is mainly solved by applying a neural network over successive sliding windows. Due to the difficulty of the task, baseline models are usually large, resulting in a high computational cost and energy consumption level. We propose a new method called SANAS (Stochastic Adaptive Neural Architecture Search) which is able to adapt the architecture of the neural network on-the-fly at inference time such that small architectures will be used when the stream is easy to process (silence, low noise, ...) and bigger networks will be used when the task becomes more difficult. We show that this adaptive model can be learned end-to-end by optimizing a trade-off between the prediction performance and the average computational cost per unit of time. Experiments on the Speech Commands dataset show that this approach leads to a high recognition level while being much faster (and/or energy saving) than classical approaches where the network architecture is static.
BibTeX
@inproceedings{veniat2019stochastic,
  author = {Tom Véniat and Olivier Schwander and Ludovic Denoyer},
  title = {{Stochastic Adaptive Neural Architecture Search for Keyword Spotting}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=AiP0IOlCzt},
}
Figure