Generating Adversarial Samples For Training Wake-up Word Detection Systems Against Confusing Words

Haoxu Wang, Yan Jia, Zeqing Zhao, Xuyang Wang, Junjie Wang, Ming Li

Abstract

Wake-up word detection models are widely used in real life, but suffer from severe performance degradation when encountering adversarial samples. In this paper we discuss the concept of confusing words in adversarial samples. Confusing words are commonly encountered, which are various kinds of words that sound similar to the predefined keywords. To enhance the wake word detection system's robustness against confusing words, we propose several methods to generate the adversarial confusing samples for simulating real confusing words scenarios in which we usually do not have any real confusing samples in the training set. The generated samples include concatenated audio, synthesized data, and partially masked keywords. Moreover, we use a domain embedding concatenated system to improve the performance. Experimental results show that the adversarial samples generated in our approach help improve the system's robustness in both the common scenario and the confusing words scenario. In addition, we release the confusing words testing database called HI-MIA-CW for future research.

Introduction

Keyword Spotting (KWS) systems, such as those used in smart speakers, must accurately identify a specific trigger phrase (e.g., "ni hao, mi ya") while ignoring all other speech. A significant challenge in real-world deployment is the presence of "confusing words" (CW)—phrases that are phonetically similar to the target keyword (e.g., "ni hao mi"). These act as natural adversarial samples, often causing high False Alarm (FA) rates.

Collecting human-recorded audio for every possible phonetic variation is expensive and time-consuming. Consequently, most training sets lack the "discriminative power" needed to reject these close matches. This paper proposes three novel data augmentation methods to generate adversarial "near-miss" samples and a domain-adaptation technique to bridge the gap between synthetic and real audio.

Proposed Methods

To improve robustness without additional human recordings, the authors introduce several generation strategies:

  1. Waveform Concatenation: Utilizing a Large Vocabulary Conversational Speech Recognition (LVCSR) system, audio from public datasets is aligned and truncated into sub-words. These segments are then stitched together to form keywords and confusing variations.
  2. Text-to-Speech (TTS) Augmentation: A multi-speaker TTS system synthesizes positive keywords, standard negative samples, and adversarial confusing words. TTS proved to be a highly effective source of adversarial data.
  3. Masked Audio: To simulate keywords interrupted by noise or silence, real keyword samples are partially masked (40%–60%) with Gaussian white noise. This teaches the model to reject incomplete or corrupted triggers.
  4. Domain Adaptation (EMB System): Synthetic and concatenated audio often have different acoustic characteristics than natural speech. To mitigate this "domain shift," the authors employ a pre-trained LSTM domain classifier to extract "domain embeddings." These embeddings are concatenated with the main model's features to help the network generalize across different audio sources.
The proposed model architecture featuring domain embedding concatenation to bridge the gap between synthetic and real speech.

Experimental Setup

The models were trained using a combination of real and generated data, including the HI-MIA dataset for real positive keywords and Aishell-1 for standard negative samples. Crucially, the authors released a new benchmark database, HI-MIA-CW, containing 16,000 recordings of 12 specific confusing word patterns from 30 speakers to evaluate performance in these difficult scenarios.

Results and Discussion

The effectiveness of the proposed methods was evaluated on a challenging test set comprising real keywords and the new "confusing words" (real+real-cw).

DET curves showing the performance of various training configurations. The combination of all augmentation methods and domain embeddings (orange line) significantly outperforms the baseline.

A baseline model trained without adversarial samples struggled significantly, showing a 68.60% False Rejection (FR) rate at a fixed false alarm threshold. By incorporating all three augmentation methods and domain embeddings (real + concat + syn + mask + EMB), the FR rate was slashed to 9.81%.

Key insights include:

  • TTS Effectiveness: TTS-generated adversarial samples provided the most substantial boost in robustness among the augmentation strategies.
  • Importance of Domain Embeddings: Simply adding synthetic data can sometimes degrade performance on real data due to the model learning simulator-specific artifacts. Domain embeddings were crucial for making synthetic data useful in real-world scenarios.
  • Discriminative Training: By explicitly showing the model "near-miss" examples, the decision boundary between "trigger" and "ignore" becomes much sharper, adhering to the principles of Maximum Mutual Information (MMI).

Conclusion

This work demonstrates that generating adversarial confusing samples is a viable and effective strategy for improving the robustness of wake-up word detection systems. The combination of TTS, waveform concatenation, and masking, supported by domain adaptation, allows models to better distinguish between keywords and phonetically similar phrases. The release of the HI-MIA-CW dataset provides a valuable resource for further research into this critical aspect of KWS performance.

Summary updated

Abstract

Wake-up word detection models are widely used in real life, but suffer from severe performance degradation when encountering adversarial samples. In this paper we discuss the concept of confusing words in adversarial samples. Confusing words are commonly encountered, which are various kinds of words that sound similar to the predefined keywords. To enhance the wake word detection system's robustness against confusing words, we propose several methods to generate the adversarial confusing samples for simulating real confusing words scenarios in which we usually do not have any real confusing samples in the training set. The generated samples include concatenated audio, synthesized data, and partially masked keywords. Moreover, we use a domain embedding concatenated system to improve the performance. Experimental results show that the adversarial samples generated in our approach help improve the system's robustness in both the common scenario and the confusing words scenario. In addition, we release the confusing words testing database called HI-MIA-CW for future research.
BibTeX
@misc{wang2022generating,
  author = {Haoxu Wang and Yan Jia and Zeqing Zhao and Xuyang Wang and Junjie Wang and Ming Li},
  title = {{Generating Adversarial Samples For Training Wake-up Word Detection Systems Against Confusing Words}},
  year = {2022},
  eprint = {2201.00167},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.00167},
}
Figure