MT-HuBERT: Self-Supervised Mix-Training for Few-Shot Keyword Spotting in Mixed Speech

Junming Yuan, Ying Shi, Dong Wang, Lantian Li, Askar HamdullaarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces MT-HuBERT, a novel self-supervised learning approach that trains AI models to identify keywords even when multiple people are speaking simultaneously. By adapting a powerful pre-training technique called Mix-Training to the HuBERT architecture, MT-HuBERT significantly improves keyword spotting performance in challenging mixed-speech scenarios, especially when labeled data is scarce.

What Problem Does This Paper Address?

Keyword spotting (KWS) systems, like the "Hey Google" or "Alexa" commands, are designed to detect specific words. While these systems have become quite good in clean environments, real-world audio is often messy. Imagine trying to activate your smart device when multiple people are talking over each other – this is the "mixed-speech scenario." Detecting a specific keyword in such a cacophony is incredibly difficult. Existing KWS models, even those using advanced self-supervised learning (SSL) techniques, often struggle with this. The challenge is to build models that can not only learn general speech patterns but also disentangle and identify individual keywords within overlapping speech, all while requiring minimal labeled examples for new keywords (few-shot learning).

Methods

The core innovation here is MT-HuBERT, which integrates the Mix-Training (MT) principle directly into the self-supervised pre-training phase of the HuBERT model.

Here's a breakdown:

  • HuBERT (Self-Supervised Learning Backbone): HuBERT is a powerful SSL model that learns speech representations by predicting masked acoustic units from context. It's pre-trained on vast amounts of unlabeled speech.
  • Mix-Training (MT): Previously proposed for supervised KWS, MT is a technique that explicitly handles mixed speech. It involves creating artificial mixtures of clean speech and training the model to detect all keywords present in the mixture, regardless of their overlap.
  • MT-HuBERT's Novelty: Instead of applying MT only during the fine-tuning (adaptation) phase, MT-HuBERT incorporates MT during the pre-training of HuBERT. This means the model learns to disentangle clean acoustic units from mixed speech from the very beginning.
  • Key Idea: The model is trained to predict "clean acoustic units" that constitute the mixture, rather than just memorizing the patterns of the mixed sound. This is inspired by how humans process complex auditory scenes.
  • Mechanism: For an n-source mixture, MT-HuBERT generates "n-hot targets" (indicating the presence of multiple units) and uses a masked prediction scheme with a multi-label binary cross-entropy objective. This forces the model to learn a compositional representation of mixed speech.
  • Adaptation: After pre-training, the MT-HuBERT backbone is frozen, and two linear layers are added for few-shot KWS adaptation. The paper explores different adaptation strategies: "Clean" (training only on clean speech), "Mixup" (linearly interpolating waveforms and labels), and "MT" (using Mix-Training during adaptation).

Datasets

  • Pre-training: LibriSpeech-960h corpus. Mixtures were limited to 2-mix, and clean utterances were interleaved.
  • Fine-tuning and Evaluation: Google Speech Commands (GSC v2) dataset.
  • Fine-tuning: Used the GSC v2 training split under three few-shot conditions: 15-shot, 5-shot, and 3-shot (meaning 15, 5, or 3 labeled examples per keyword).
  • Testing: * Clean Test: Official GSC v2 test set.
  • Mixed Speech Test: Custom-constructed 2-mix and 3-mix versions by randomly mixing clean-test utterances. The 3-mix set was specifically used to evaluate generalization to unseen mixture counts.

Key Insights and Ideas

  • Pre-training Matters Most: Integrating Mix-Training into the pre-training phase (MT-HuBERT) is more impactful than just using Mix-Training during adaptation. This suggests that learning to handle mixtures early on is crucial for robust performance.
  • Compositionality Over Memorization: The hypothesis that representing mixed speech as a composition of clean units (MT-HuBERT's approach) leads to better generalization than memorizing mixed patterns (like in MPC) is strongly supported by the results. This aligns with human auditory perception.
  • MT is a General Solution: The Mix-Training adaptation strategy consistently outperforms "Clean" or "Mixup" adaptation, highlighting MT's effectiveness for mixed-speech processing across various models.
  • Unseen Complexity is Manageable: MT-HuBERT shows remarkable generalization to 3-mix scenarios even though it was only pre-trained on 2-mix data. This indicates the learned compositional representations are robust.

Key Results

MT-HuBERT, when combined with MT adaptation, consistently achieved state-of-the-art performance across all tested scenarios (clean, 2-mix, and 3-mix) and shot counts (15, 5, and 3).

  • Clean Speech: MT-HuBERT achieved the highest accuracy (e.g., 93.80% ACC at 15-shot).
  • 2-Mix Speech: MT-HuBERT significantly outperformed competitors, achieving 79.78% ACC (15-shot), a substantial gain over its closest competitor (Cocktail-HuBERT + MT) by +1.24% ACC.
  • 3-Mix Speech (Unseen): This is where MT-HuBERT truly shone. It achieved 65.91% ACC (15-shot), outperforming Cocktail-HuBERT + MT by +3.26%. This demonstrates strong generalization to more complex mixtures than seen during training.
  • Low-Shot Performance: The advantage of MT-HuBERT was more pronounced in lower-shot regimes (5-shot and 3-shot), indicating its efficiency in data-scarce situations.

Limitations and Future Work

  • Limitations:
    • The pre-training was primarily conducted on 2-mix data, and the evaluation extended to 3-mix. While generalization was good, performance might degrade with even higher mixture cardinalities.
  • The paper doesn't explicitly discuss performance under varying Signal-to-Noise Ratios (SNR).
  • Future Work: The authors suggest several promising directions:
  • Scaling mixture cardinality and SNR diversity during pre-training.
  • Extending the approach to larger and multilingual corpora.
  • Evaluating the transferability of MT-HuBERT to other speech tasks, such as Automatic Speech Recognition (ASR).

Summary updated

BibTeX
@misc{yuan2025mt,
  author = {Junming Yuan and Ying Shi and Dong Wang and Lantian Li and Askar Hamdulla},
  title = {{MT-HuBERT: Self-Supervised Mix-Training for Few-Shot Keyword Spotting in Mixed Speech}},
  year = {2025},
  eprint = {2511.06296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.06296},
}
Figure