Open-Vocabulary Keyword-Spotting with Adaptive Instance Normalization

Aviv Navon, Aviv Shamsian, Neta Glazer, Gill Hetz, Joseph Keshet

TL;DR

AdaKWS is a novel open-vocabulary keyword-spotting (KWS) framework that moves away from traditional joint audio-text embedding spaces. Instead, it uses a text encoder to generate keyword-specific normalization parameters that "re-style" a transformer-based audio classifier via Adaptive Instance Normalization (AdaIN), achieving state-of-the-art accuracy and up to 160x faster inference than heavy-duty ASR baselines.

What Problem Does This Paper Address?

Keyword spotting (KWS) is the task of detecting specific words (like "Hey Siri") in a stream of audio. Traditional systems are "closed-vocabulary," meaning they can only detect words they were explicitly trained on. To detect a new word, you usually have to retrain the model.

"Open-vocabulary" KWS aims to detect any user-defined keyword. Current approaches typically map audio and text into a shared latent space to find matches. However, these methods often suffer from:

  1. Representation Misalignment: The audio and text encoders are often disjoint, making it hard to project them perfectly into the same space.
  2. Phoneme Dependency: Many models rely on phoneme dictionaries, which are unavailable for many low-resource languages.
  3. Efficiency: High-accuracy models (like Whisper) are often too slow for real-time, low-latency KWS applications.

Methods

The authors introduce AdaKWS, which treats the keyword as a "style" that conditions the audio processing.

  • Keyword-Conditioned Normalization: Instead of a joint embedding, they use a character-based LSTM text encoder to map a keyword into a set of parameters ($\mu$ and $\sigma$).
  • Adaptive Instance Normalization (AdaIN): These parameters are injected into a transformer-based audio classifier. Specifically, they replace standard Layer Normalization with AdaIN layers. This effectively "tunes" the audio processor to be sensitive to the specific acoustic patterns of the target keyword.
  • Frozen Audio Backbone: To keep the model robust, they use a pre-trained, frozen Whisper transformer encoder as the primary audio feature extractor.
  • Hard Negative Mining: To prevent the model from getting confused by similar-sounding words (e.g., "cat" vs. "bat"), they introduce a training strategy that specifically samples "hard negatives" using character substitution, keyword concatenation, and nearest-neighbor lookups in the text embedding space.

Datasets

The model was evaluated across a diverse set of multilingual and open-vocabulary benchmarks:

  • VoxPopuli: A massive multilingual speech corpus (16 languages) used for primary training and testing.
  • LibriPhrase: A benchmark specifically for open-vocabulary KWS, featuring "Easy" and "Hard" (acoustically similar) splits.
  • Multilingual LibriSpeech (MLS): Used to test zero-shot generalization.
  • Fleurs: Used to evaluate performance on low-resource languages (Icelandic, Maltese, Swahili, Uzbek) that were unseen during training.

Key Insights and Ideas

  • KWS as Style Transfer: The most creative leap is applying AdaIN—a technique famous in computer vision for style transfer—to audio. By treating the keyword as the "style," the model learns to filter the audio signal specifically for that word's signature.
  • End-to-End Sentence Training: Unlike many KWS models that require perfectly clipped audio of single words, AdaKWS is trained on entire sentences (up to 30 seconds). This removes the need for tedious word-level alignment and allows the model to learn from more natural, context-rich data.
  • The Power of Hard Negatives: The ablation study shows that simple random negative sampling isn't enough. Incorporating "Nearest Keyword" sampling (finding words that look/sound similar) was the single biggest factor in boosting the F1 score from ~79 to ~92.

Key Results

  • Superior Accuracy: AdaKWS outperformed Whisper-Large-v2 (a much larger model) on the VoxPopuli dataset by over 6% in F1 score.
  • State-of-the-Art on LibriPhrase: On the "Hard" split of LibriPhrase, AdaKWS-Small achieved an AUC of 95.09%, significantly beating previous SOTA methods like CED (92.70%).
  • Massive Speedup: In zero-shot tests on Multilingual LibriSpeech, the AdaKWS-Small model achieved comparable performance to Whisper-Large-v2 while being 160x faster at inference time.
  • Cross-Lingual Generalization: The model showed strong "zero-shot" capabilities, performing well on low-resource languages like Maltese and Swahili without any language-specific fine-tuning.

Limitations and Future Work

  • Reliance on Audio Backbone: While using a frozen Whisper encoder provides a strong foundation, the model's performance is inherently tied to the quality of that pre-trained backbone.
  • Character-Set Constraints: The current text encoder is character-based. While it generalizes well, performance might vary significantly for languages with vastly different scripts (e.g., logographic systems like Chinese) if not represented in the training set.
  • Future Directions: The authors suggest that this adaptive normalization approach could be extended to other speech tasks beyond KWS, potentially allowing for highly efficient, user-conditioned speech recognition or speaker identification.

Summary updated

Abstract

Open vocabulary keyword spotting is a crucial and challenging task in automatic speech recognition (ASR) that focuses on detecting user-defined keywords within a spoken utterance. Keyword spotting methods commonly map the audio utterance and keyword into a joint embedding space to obtain some affinity score. In this work, we propose AdaKWS, a novel method for keyword spotting in which a text encoder is trained to output keyword-conditioned normalization parameters. These parameters are used to process the auditory input. We provide an extensive evaluation using challenging and diverse multi-lingual benchmarks and show significant improvements over recent keyword spotting and ASR baselines. Furthermore, we study the effectiveness of our approach on low-resource languages that were unseen during the training. The results demonstrate a substantial performance improvement compared to baseline methods.
BibTeX
@inproceedings{navon2024open,
  author = {Aviv Navon and Aviv Shamsian and Neta Glazer and Gill Hetz and Joseph Keshet},
  title = {{Open-Vocabulary Keyword-Spotting with Adaptive Instance Normalization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=U86knm36w9},
}

Loading related content...

Figure