Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding

Kumari Nishu, Minsik Cho, Paul Dixon, Devang Naik

Introduction

Keyword spotting (KWS) systems are integral to modern human-computer interaction, typically categorized into fixed and flexible systems. While fixed KWS (e.g., "Hey Siri") relies on pre-defined triggers, flexible KWS allows users to define custom trigger phrases on the fly. However, implementing flexible KWS on low-power edge devices presents a significant challenge: the "modality gap."

Audio signals are continuous and acoustic, whereas text is discrete and symbolic. Traditional approaches attempt to bridge this gap using separate, large-scale encoders for each modality, often requiring complex projection layers to align their respective embedding spaces. Researchers at Apple have proposed a more efficient solution: the Common Embedding based Detector (CED), which achieves state-of-the-art performance by creating a homogeneous, audio-compliant embedding space.

Bridging the Modality Gap

The fundamental innovation of the CED model is the shift from forcing alignment via mathematical projections to building text embeddings directly within the audio encoder's latent space. In traditional systems, an audio encoder and a text encoder (such as DistilBERT) are trained independently, resulting in a "Large Mismatch" between their outputs. Aligning these requires significant computational overhead and massive models that are unsuitable for "always-on" hardware.

Conceptual comparison between traditional projection-based alignment and the proposed audio-compliant embedding space.

By contrast, the CED approach aims for a "Small Mismatch" by making the text encoder "audio-compliant." Instead of treating text as a linguistic entity, the system treats it as a sequence of phonetic sounds, ensuring that both modalities share the same mathematical "language" from the start.

The CED Architecture

The CED architecture is designed for both high accuracy and extreme efficiency, replacing heavy language models with a lightweight phonetic mapping system. The pipeline consists of three main modules: the Audio Encoder, the Audio-compliant Text Encoder, and the Verifier.

Detailed overview of the Common Embedding based Detector (CED) system architecture.

Audio-Compliant Text Encoder

Rather than using a 66-million parameter transformer like DistilBERT, the CED uses a 0.83-million parameter Grapheme-to-Phoneme (G2P) model. This model converts input text into phonemes, which are then mapped to vectors using a Phoneme-to-Vector (P2V) lookup table. These vectors are not learned from text data; instead, they are centroids extracted directly from the audio encoder's latent space during a preliminary training phase. This ensures that the resulting text embedding is inherently aligned with how those phonemes sound in the audio domain.

The Verifier and DSP

The Verifier module computes a cosine similarity matrix between the audio and text embeddings. To handle the temporal variations in speech, it employs Dynamic Sequence Partitioning (DSP). DSP finds the optimal alignment between the continuous audio stream and the discrete phoneme sequence. The aligned data is then processed by a Gated Recurrent Unit (GRU) and a Fully Connected (FC) layer to determine if a keyword match has occurred.

Training and Robustness

To ensure the model can distinguish between similar-sounding words, the researchers introduced a "confusable keyword generation" scheme. During training, the system generates "hard negatives"—words that are phonetically similar to the target keyword (e.g., "stamp" vs. "stop"). This phonetic stress test forces the model to attend to fine-grained acoustic details, significantly reducing false-positive triggers.

Experimental Results

The CED model was evaluated against industry-standard benchmarks, demonstrating superior performance and generalization:

  • Libriphrase Hard Dataset: The model achieved an Area Under the ROC Curve (AUC) of 92.7%, a substantial increase from the previous baseline of 84.21%. The Equal-Error-Rate (EER) dropped from 23.36% to 14.4%.
  • Google Speech Commands V1: To test zero-shot generalization, the model was evaluated without fine-tuning, resulting in a 50.6% relative improvement in EER compared to existing state-of-the-art methods.
  • Efficiency: By utilizing the G2P and P2V approach, the system reduced text-processing complexity by approximately 80x compared to DistilBERT-based solutions.

Conclusion

The CED framework demonstrates that homogeneity is more effective than projection for cross-modal alignment in KWS. By using phonemes as a universal bridge and extracting embeddings directly from the audio space, the researchers created a system that is both more accurate and significantly more efficient. Future work may involve further optimizing the "Tiny Conformer" audio encoder and refining the P2V centroids to better handle diverse accents and noisy environments, paving the way for more robust, user-defined trigger words on edge devices.

Summary updated

BibTeX
@inproceedings{nishu2024flexible,
  author = {Kumari Nishu and Minsik Cho and Paul Dixon and Devang Naik},
  title = {{Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447547},
}
Figure