Separable Temporal Convolution plus Temporally Pooled Attention for Lightweight High-performance Keyword Spotting

Shenghua Hu, Jing Wang, Yujun Wang, Wenjing Yang

Abstract: Keyword spotting (KWS) on mobile devices generally requires a small memory footprint. However, most current models still maintain a large number of parameters in order to ensure good performance. In this paper, we propose a temporally pooled attention module which can capture global features better than the AveragePool. Besides, we design a separable temporal convolution network which leverages depthwise separable and temporal convolution to reduce the number of parameter and calculations. Finally, taking advantage of separable temporal convolution and temporally pooled attention, a efficient neural network (ST-AttNet) is designed for KWS system. We evaluate the models on the publicly available Google speech commands data sets V1. The number of parameters of proposed model (48K) is 1/6 of state-of-the-art TC-ResNet14-1.5 model (305K). The proposed model achieves a 96.6% accuracy, which is comparable to the TC-ResNet14-1.5 model (96.6%).

The Challenge of Always-On KWS

Keyword spotting (KWS) is the foundational technology for voice-activated assistants. For these systems to be effective, they must remain "always-on," monitoring audio streams for specific trigger phrases like "Hey Siri" or "OK Google." This requirement imposes strict constraints on mobile hardware: the model must be highly accurate to minimize false positives while maintaining a minimal memory footprint and low power consumption.

While deep learning architectures like ResNet have significantly improved KWS accuracy, they often come with a high computational cost—hundreds of thousands of parameters and millions of operations. This paper introduces ST-AttNet, a lightweight model designed to bridge the gap between high performance and extreme resource efficiency.

Proposed Architecture: ST-AttNet

The ST-AttNet architecture leverages two key innovations to achieve its efficiency: Separable Temporal Convolutions and Temporally Pooled Attention.

1. Separable Temporal Convolution

Traditional KWS models often treat audio spectrograms as 2D images. ST-AttNet instead utilizes 1D convolutions that operate exclusively along the time axis. To further reduce complexity, the authors employ depthwise separable convolutions. This technique decomposes a standard convolution into a depthwise convolution (filtering each input channel separately) and a pointwise convolution (combining the results). This approach drastically reduces the number of parameters and calculations required.

2. Temporally Pooled Attention

Most KWS models use simple Average Pooling to aggregate temporal features before classification. However, speech signals are non-stationary; certain segments of a keyword are more discriminative than others. The proposed Temporally Pooled Attention mechanism learns to assign weights to different time steps globally, allowing the network to focus on the most informative acoustic cues.

The architecture of ST-AttNet, highlighting the separable temporal convolution blocks and the overall model flow.

3. Dilated Convolutions

To capture long-term temporal dependencies without significantly increasing the number of layers, the model incorporates dilated convolutions. By increasing the "gap" between kernel elements, the receptive field expands exponentially, allowing the model to "see" a wider window of audio data efficiently.

Experimental Evaluation

The researchers evaluated ST-AttNet on the Google Speech Commands Dataset (V1), which contains over 64,000 recordings of 30 different words. The task involved a standard 12-class classification (10 keywords plus "unknown" and "silence"). Audio was preprocessed into 40-dimensional Mel-Frequency Cepstrum Coefficient (MFCC) frames.

Key Results

ST-AttNet demonstrates exceptional performance relative to its size. The "wide" version of the model achieved 96.6% accuracy, matching the state-of-the-art TC-ResNet14-1.5 model. Crucially, ST-AttNet-wide achieves this with only 48K parameters—approximately 1/6th the size of TC-ResNet14-1.5 (305K parameters).

ROC curve comparing the false reject rate and false alarm rate of ST-AttNet4-wide against other baseline models.

When compared to the popular Res15 model, ST-AttNet4 achieved superior performance with a 10x reduction in parameters and a 447x reduction in multiply operations. These results highlight the effectiveness of combining temporal attention with separable convolutions for edge-based audio processing.

Conclusion

ST-AttNet represents a significant step forward in lightweight keyword spotting. By replacing generic pooling with learned attention and optimizing convolutional operations for the temporal domain, the model achieves state-of-the-art accuracy with a fraction of the traditional resource requirements. This makes it an ideal candidate for next-generation, energy-efficient "always-on" mobile applications.

Summary updated

Abstract

Keyword spotting (KWS) on mobile devices generally requires a small memory footprint. However, most current models still maintain a large number of parameters in order to ensure good performance. In this paper, we propose a temporally pooled attention module which can capture global features better than the AveragePool. Besides, we design a separable temporal convolution network which leverages depthwise separable and temporal convolution to reduce the number of parameter and calculations. Finally, taking advantage of separable temporal convolution and temporally pooled attention, a efficient neural network (ST-AttNet) is designed for KWS system. We evaluate the models on the publicly available Google speech commands data sets V1. The number of parameters of proposed model (48K) is 1/6 of state-of-the-art TC-ResNet14-1.5 model (305K). The proposed model achieves a 96.6% accuracy, which is comparable to the TC-ResNet14-1.5 model (96.6%).
BibTeX
@misc{hu2021separable,
  author = {Shenghua Hu and Jing Wang and Yujun Wang and Wenjing Yang},
  title = {{Separable Temporal Convolution plus Temporally Pooled Attention for Lightweight High-performance Keyword Spotting}},
  year = {2021},
  eprint = {2108.12146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.12146},
}
Figure