A Separable Temporal Convolution Neural Network with Attention for Small-Footprint Keyword Spotting

Shenghua Hu, Jing Wang, Yujun Wang, Lidong Yang, Wenjing Yang

Abstract

Keyword spotting (KWS) on mobile devices generally requires a small memory footprint. However, most current models still maintain a large number of parameters in order to ensure good performance. To solve this problem, this paper proposes a separable temporal convolution neural network with attention (ST-Conv), which utilizes a small number of parameters. Through temporal convolution combined with an attention mechanism, a model with only 32.2K parameters is implemented while maintaining high performance. The proposed model achieves 95.7% accuracy on the Google Speech Commands dataset, which is close to the performance of Res15 (239K), the current state-of-the-art model in KWS.

Introduction

Keyword spotting—the technology enabling voice-activated features like "Hey Siri" or "OK Google"—faces a persistent challenge in balancing accuracy with efficiency. To be effective, these models must operate reliably in noisy environments. However, because they run locally on mobile and IoT devices to ensure privacy and minimize battery consumption, they must have a minimal memory footprint and low computational requirements.

Traditional approaches often employ 2D Convolutional Neural Networks (CNNs) adapted from image processing. While effective, these models treat audio spectrograms as static images, which can be computationally inefficient. Such architectures often require hundreds of thousands of parameters to achieve high accuracy, making them unsuitable for highly constrained hardware environments.

Methodology: The ST-Conv Architecture

The researchers introduce ST-Conv (Separable Temporal Convolution), an architecture designed to optimize the processing of audio features (specifically Mel-frequency cepstral coefficients, or MFCCs) through four key innovations:

  1. Frequency-First Extraction: Unlike standard 2D filters, the initial layer employs a $1 \times 40$ convolution. This collapses the frequency domain into a single dimension, allowing subsequent layers to focus exclusively on temporal patterns.
  2. Depthwise Separable Temporal Convolutions: The model utilizes $3 \times 1$ kernels instead of $3 \times 3$. This transition from 2D to 1D temporal convolutions reduces the parameter count by two-thirds per layer without compromising the model's ability to learn speech patterns.
  3. Exponentially Dilated Convolutions: To achieve a wide receptive field (121 frames) using only 13 layers, the authors use dilated convolutions where the dilation rate increases exponentially ($d = 2^{\lfloor i/3 \rfloor}$).
  4. Hybrid Backend: The CNN stages are followed by a Bidirectional Gated Recurrent Unit (BGRU) to capture long-term dependencies and a Shared Weight Self-Attention (SWSA) mechanism. The attention layer allows the model to focus on the most discriminative phonemes within the audio signal.
Architecture of the ST-Conv model, showing (a) the separable convolution block, (b) the overall model pipeline, and (c) the self-attention (SWSA) mechanism.

Experimental Setup

The model was evaluated using the Google Speech Commands V1 dataset, which contains 64,752 one-second recordings of 30 different words. The task involved recognizing 10 specific keywords (e.g., "yes", "no", "up", "down") while treating the remaining 20 words as "fillers" or background noise. The dataset was split into standard training (51K), development (6.8K), and test (6.8K) sets.

Results and Discussion

The ST-Conv model demonstrates an exceptional balance between size and performance. It achieved a test accuracy of 95.7%, matching the performance of the state-of-the-art Res15 model while being nearly 8 times smaller in terms of parameter count.

Comparison of test accuracy, parameter count (Par.), and multiplier operations (Mult.) across various KWS models.

Key insights from the results include:

  • Efficiency: ST-Conv requires only 32.2K parameters and 3.09 million multipliers, making it significantly more energy-efficient than traditional Att-RNN models (22.6M Mults).
  • Scalability: A "Narrow" version of the model, containing only 9.4K parameters, still maintained a high accuracy of 94.3%.
  • Impact of Attention: Replacing standard average pooling with the SWSA layer resulted in a 0.8% accuracy boost, confirming that self-attention effectively identifies critical segments of the keyword signal.

Conclusion

ST-Conv proves that high-accuracy keyword spotting does not require massive parameter counts. By prioritizing temporal features and utilizing efficient separable convolutions, the model achieves state-of-the-art results with a tiny footprint. Future work will focus on optimizing the architecture for continuous streaming audio and evaluating power consumption on specific microcontroller hardware.

Summary updated

Abstract

Keyword spotting (KWS) on mobile devices generally requires a small memory footprint. However, most current models still maintain a large number of parameters in order to ensure good performance. To solve this problem, this paper proposes a separable temporal convolution neural network with attention, it has a small number of parameters. Through the time convolution combined with attention mechanism, a small number of parameters model (32.2K) is implemented while maintaining high performance. The proposed model achieves 95.7% accuracy on the Google Speech Commands dataset, which is close to the performance of Res15(239K), the state-of-the-art model in KWS at present.
BibTeX
@misc{hu2021separable,
  author = {Shenghua Hu and Jing Wang and Yujun Wang and Lidong Yang and Wenjing Yang},
  title = {{A Separable Temporal Convolution Neural Network with Attention for Small-Footprint Keyword Spotting}},
  year = {2021},
  eprint = {2109.00260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.00260},
}
Figure