Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder

Alexandre Bittar, Paul Dixon, Mohammad Samragh, Kumari Nishu, Devang Naik

Abstract

Keyword spotting (KWS) systems for "always-on" devices face a strict trade-off between computational efficiency and accuracy. This paper introduces a streaming Conformer encoder that utilizes Input-Dependent Dynamic Depth (I3D) to skip unnecessary computations during non-speech or irrelevant audio segments. By treating audio as a 1D image and applying vision-inspired localization techniques, the proposed model achieves higher accuracy than traditional convolutional baselines while reducing Multiply-Accumulate (MAC) operations by up to 97% during silence.

Introduction

Modern voice assistants rely on "always-on" keyword spotting to detect trigger phrases like "Hey Siri." These systems must operate continuously on battery-powered devices, necessitating extremely low power consumption and a small memory footprint. However, they must also remain robust against background noise and diverse acoustic environments.

While state-of-the-art Automatic Speech Recognition (ASR) models, such as the Conformer, offer superior accuracy through a combination of self-attention and convolutions, their computational cost is typically prohibitive for continuous edge-device deployment. Furthermore, standard architectures process every audio frame with equal intensity, wasting energy on silence or background noise. This research addresses these challenges by integrating dynamic module skipping into a streaming Conformer architecture.

Architecture and Methodology

The proposed pipeline treats the audio signal as a 1D image, enabling the use of vision-inspired detection and localization techniques.

The overall system architecture includes input processing, a Conformer encoder with I3D gating, and multi-head output layers for detection, classification, and localization.

Streaming Conformer Encoder

The backbone of the system is a Conformer encoder, which captures both long-range context via self-attention and local patterns via convolutions. To enable real-time operation, the model processes audio in 1.2-second windows with a 240ms shift.

Input-Dependent Dynamic Depth (I3D)

The core innovation is the introduction of trainable binary gates within the Conformer blocks. Each module (Feed-forward, Attention, and Convolution) is paired with a small linear layer that decides whether to execute or skip the module based on the current input. During training, the Gumbel-Softmax trick is employed to learn these discrete decisions. This mechanism allows the model to act as a sophisticated Voice Activity Detector (VAD), bypassing complex layers when processing background noise or silence.

Multi-Head Output Layers

Inspired by CenterNet, the model features three specialized output heads:

  • Detection: Determines the presence of a keyword.
  • Classification: Identifies the specific keyword.
  • Localization: Predicts the precise start and end times of the keyword using a bounding box approach.

Experimental Results

The model was evaluated on the Librispeech (top 1000 words) and Google Speech Commands (GSC) v2 datasets. To simulate realistic conditions, GSC commands were overlaid with background "babble noise."

Performance comparison of the proposed Conformer models (Ours-L, Ours-S, Ours-XS) against BC-ResNet baselines across Librispeech and GSC datasets.

The results demonstrate that the Conformer-based approach significantly outperforms the standard BC-ResNet-L baseline. On the noisy Librispeech "test-other" set, the "Ours-L" model improved the F1 score from 0.715 to 0.779 while using 16% fewer parameters.

The I3D gating mechanism proved highly effective at reducing computational load. In continuous speech, the gated model skipped approximately 30% of operations. Most notably, in the GSC test with background noise, the model skipped up to 97% of computations during non-speech intervals. The model also showed improved Intersection over Union (IOU) scores, indicating higher precision in localizing keyword boundaries.

Discussion and Conclusion

The integration of dynamic depth into a streaming Conformer architecture allows for high-accuracy keyword spotting with a fraction of the typical computational cost. By learning to skip modules during silence or noise, the model preserves battery life without sacrificing performance.

However, some limitations remain. While I3D reduces active computation (MACs), the full model weights must still reside in memory, meaning the storage footprint is not reduced. Additionally, the system's latency is constrained by the 240ms window stride. Future work may explore using this gated Conformer as a low-power "first-stage" trigger in a multi-tier ASR hierarchy, further optimizing the balance between power and performance.

Summary updated

BibTeX
@inproceedings{bittar2024improving,
  author = {Alexandre Bittar and Paul Dixon and Mohammad Samragh and Kumari Nishu and Devang Naik},
  title = {{Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447485},
}
Figure