Understanding temporally weakly supervised training: A case study for keyword spotting

Heinrich Dinkel, Weiji Zhuang, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun WangarXiv 2023 · DOI · arXiv

Abstract

Keyword spotting (KWS) traditionally relies on "strong supervision"—precise frame-level labels indicating exactly when a keyword starts and ends. However, obtaining such labels is labor-intensive and prone to errors, especially in noisy conditions. This paper investigates "weak supervision," where models are trained on multi-second noisy clips labeled only with the keyword's presence. We demonstrate that an end-to-end (E2E) model using a TC-ResNet8 architecture can implicitly learn to localize keywords, achieving performance comparable to or better than strongly supervised "Oracle" baselines. Furthermore, the weakly supervised approach shows significantly greater robustness to noise and label misalignment compared to traditional forced-alignment methods.

Introduction

Modern voice assistants rely on KWS to trigger wake-up functions. Training these models usually necessitates strong labels, often generated via "forced alignment" (FA) from existing speech recognition systems. Unfortunately, FA frequently fails in noisy environments, producing "dirty" labels that degrade model performance. This research explores whether we can bypass granular labeling by using "weak labels" (clip-level tags) without sacrificing accuracy. The central question is whether a neural network can learn to localize a keyword buried within several seconds of background noise.

Methodology

The study compares two paradigms: a strongly supervised baseline using a Time-Delay Neural Network (TDNN) trained with Lattice-Free Maximum Mutual Information (LF-MMI), and a weakly supervised E2E classifier using a TC-ResNet8 architecture. The E2E model is only informed of the keyword's identity, not its temporal location.

The core innovation is "temporally weak" training. As illustrated in Figure 1, 1-second keyword samples are injected into longer (3s, 5s, or 7s) noisy segments. To ensure generalization, the researchers employ random cropping during training, forcing the network to identify the keyword regardless of its position within the audio window. This matching of the training crop size to the expected keyword length is a vital heuristic for success.

Figure 1: Training data preparation process, showing how 1-second keywords are randomly inserted into longer noisy clips to create weakly supervised training samples.

Experimental Setup

The experiments utilize two primary datasets: the Google Speech Commands V1 (GSCV1) for core keyword utterances and AudioSet for unconstrained real-world background noise. These are combined to create three environments:

  • Clean: The original 1-second GSCV1 samples.
  • Weak-ts: Keywords randomly inserted into $t$-second noise clips without overlapping the noise.
  • WeakSNR-ts: Keywords mixed directly with noise at specific Signal-to-Noise Ratios (0, 5, and 10 dB).

Results and Discussion

The weakly supervised E2E model demonstrated remarkable performance and robustness. On clean data, it achieved 97.03% accuracy, slightly outperforming the strongly supervised Oracle baseline (96.23%).

A critical finding is the model's resilience to clip duration and noise. While the Force-Align baseline's accuracy plummeted to 87.21% on 7-second clips due to poor label quality, the E2E model maintained a high accuracy of 96.68% (see Figure 2).

Figure 2: Performance comparison between Oracle, Force-Align (FA), and the proposed weakly supervised E2E model across different clip durations.

Key insights from the study include:

  • Implicit Localization: The model "automatically" learns to ignore surrounding noise and focus on relevant features without explicit boundary information.
  • The 15% Threshold: Weakly supervised models tend to outperform Oracle baselines as long as the keyword occupies at least 15% of the total clip duration.
  • Regularization Effect: Weak labels act as a form of label smoothing, helping the model generalize better to real-world scenarios where boundaries are often fuzzy.

Conclusion

This work demonstrates that temporally weak supervision is a viable and often superior alternative to strong supervision for KWS. By matching the training crop size to the expected keyword length, models can achieve high accuracy and robustness without the need for expensive or unreliable frame-level annotations. Future research directions include applying these principles to other tasks like Sound Event Detection (SED) or Voice Activity Detection (VAD), where temporal boundaries are similarly difficult to define.

Summary updated

BibTeX
@misc{dinkel2023understanding,
  author = {Heinrich Dinkel and Weiji Zhuang and Zhiyong Yan and Yongqing Wang and Junbo Zhang and Yujun Wang},
  title = {{Understanding temporally weakly supervised training: A case study for keyword spotting}},
  year = {2023},
  eprint = {2305.18794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.18794},
}
Figure