Abstract
Open vocabulary keyword spotting is a crucial and challenging task in automatic speech recognition (ASR) that focuses on detecting user-defined keywords within a spoken utterance. Keyword spotting methods commonly map the audio utterance and keyword into a joint embedding space to obtain some affinity score. In this work, we propose AdaKWS, a novel method for keyword spotting in which a text encoder is trained to output keyword-conditioned normalization parameters. These parameters are used to process the auditory input. We provide an extensive evaluation using challenging and diverse multi-lingual benchmarks and show significant improvements over recent keyword spotting and ASR baselines. Furthermore, we study the effectiveness of our approach on low-resource languages that were unseen during the training. The results demonstrate a substantial performance improvement compared to baseline methods.
Introduction
Keyword spotting (KWS) is the task of detecting specific words, such as wake words, in a stream of audio. Traditional systems are typically "closed-vocabulary," requiring retraining to recognize new terms. In contrast, "open-vocabulary" KWS aims to detect any user-defined keyword. Current state-of-the-art approaches often map audio and text into a shared latent space. However, these methods frequently suffer from representation misalignment between disjoint encoders, a heavy dependency on phoneme dictionaries (which limits low-resource language support), and high computational latency that precludes real-time application.
Methodology
AdaKWS introduces a paradigm shift by treating the keyword as a "style" that conditions audio processing. Instead of seeking a joint embedding, the framework uses a character-based LSTM text encoder to map a keyword $v$ into a set of normalization parameters $\theta_v = (\mu_v, \sigma_v)$.
These parameters are injected into a transformer-based audio classifier via Adaptive Instance Normalization (AdaIN). By replacing standard Layer Normalization with AdaIN, the model effectively "tunes" the audio processor to be sensitive to the specific acoustic patterns of the target keyword. To ensure robustness, AdaKWS utilizes a frozen pre-trained Whisper transformer encoder as its primary feature extractor. Furthermore, the model is trained using a "Hard Negative Mining" strategy, which samples acoustically similar distractors to prevent confusion between similar-sounding words.
Experimental Results
The model was evaluated across several multilingual benchmarks, including VoxPopuli, LibriPhrase, and Multilingual LibriSpeech (MLS).
As shown in the performance comparison, AdaKWS-Small achieved an overall F1 score of 94.6% on the VoxPopuli dataset, significantly outperforming the much larger Whisper-Large-V2 (88.4%). On the challenging "Hard" split of LibriPhrase, AdaKWS-Small reached a state-of-the-art AUC of 95.09%. Beyond accuracy, the model is highly efficient; in zero-shot tests, AdaKWS-Small achieved comparable performance to Whisper-Large-V2 while being approximately 160x faster at inference time.
Conclusion
AdaKWS demonstrates that keyword-conditioned normalization via AdaIN is a highly effective alternative to traditional joint embedding spaces for open-vocabulary KWS. The approach provides superior accuracy, exceptional inference speed, and strong zero-shot generalization to low-resource languages like Maltese and Swahili. Future work may explore extending this adaptive normalization technique to broader speech tasks such as user-conditioned recognition or speaker identification.
Summary updated