Abstract
Keyword spotting (KWS) is a critical component for always-on voice interfaces, typically requiring a neural network to process continuous audio streams. However, standard static architectures often waste significant energy processing silence or background noise. We present Stochastic Adaptive Neural Architecture Search (SANAS), a framework that dynamically adjusts a neural network's architecture at inference time. By using a recurrent policy to sample sub-networks based on the audio context, SANAS allocates high computational resources to complex speech segments and minimal resources to background noise. Our experiments on the Speech Commands dataset demonstrate that SANAS achieves state-of-the-art accuracy while reducing computational costs by up to 70% compared to static baselines.
Introduction
Keyword spotting (KWS) systems, such as those used for "Hey Siri" or "OK Google," must operate continuously under strict power constraints. Traditionally, this is achieved by sliding a fixed-size neural network over an audio stream. This approach is inherently inefficient because the computational cost remains constant regardless of whether the input is a complex keyword, simple background noise, or silence.
Existing Neural Architecture Search (NAS) methods focus on finding a single optimal static architecture. In contrast, we propose a dynamic approach where the model's complexity adapts to the difficulty of the input signal. This "on-the-fly" adaptation allows for significant energy savings on mobile and IoT devices without sacrificing recognition performance.
The SANAS Framework
SANAS treats architecture selection as a dynamic, recurrent process. The framework is built upon a "Super-Network"—a large graph containing numerous possible sub-graphs of varying depths and widths. In our implementation, we utilize the popular cnn-trad-fpool3 architecture as a base and augment it with multiple shortcut connections to create a rich search space.
At each timestep $t$, a Recurrent Neural Network (RNN) maintains a latent state $z_t$ representing the audio context. This RNN generates a distribution over possible architectures, from which a specific sub-graph is sampled. To train this non-differentiable sampling process, we employ Monte-Carlo gradient estimation (similar to the REINFORCE algorithm) with variance reduction. The model is optimized using a budgeted loss function that balances prediction accuracy against the average computational cost (FLOPs).
Context-Aware Resource Allocation
A key insight of SANAS is its ability to learn word-specific and noise-specific computational budgets. During the training process, the model naturally discovers that background noise requires far less "thinking time" than actual speech.
As shown in the figure above, the FLOP count for background noise frames (the orange line) drops dramatically as training progresses, eventually settling at a fraction of the cost required for keywords. This demonstrates that the model successfully learns to "wake up" only when the input signal contains potentially relevant information.
Experimental Results
We evaluated SANAS on the Speech Commands dataset, simulating a real-world streaming environment by mixing background noise with randomly located words. The results indicate that dynamic architectures consistently outperform static ones in terms of the accuracy-cost trade-off.
As illustrated in the trade-off plot, SANAS configurations (green circles) occupy a superior Pareto front compared to the static cnn-trad-fpool3 baselines. For instance, one SANAS variant achieved an 86.5% match rate using only 37.7M FLOPs per frame, whereas the standard static baseline required 124.6M FLOPs to achieve a lower 81.7% match rate. In its most aggressive power-saving mode, SANAS can operate at just 519K FLOPs—a 240x reduction from the baseline—while still maintaining a functional 61.2% match rate for low-power wake-up scenarios.
Conclusion
SANAS represents a significant step toward efficient, always-on keyword spotting. By moving from static to adaptive architectures, we can achieve higher accuracy with a fraction of the computational overhead. Future work will focus on translating these theoretical FLOP savings into actual battery life improvements on specialized edge hardware and exploring even larger search spaces for more complex audio tasks.
Summary updated