Abstract
Wake-up word detection (WWD) is a critical component for voice-activated systems, requiring high sensitivity and extremely low power consumption for always-on operation. While large-scale self-supervised learning (SSL) models like wav2vec 2.0 have significantly advanced speech processing, their massive parameter counts make them unsuitable for edge devices. This paper introduces LiteFEW (Lightweight Feature Encoder for WWD), a framework designed to distill the rich representations of SSL models into a tiny footprint. By employing architecture pruning and a specialized auto-encoder-based distillation process, LiteFEW achieves over 20% relative performance improvement in False Rejection Rate (FRR) compared to traditional Mel-filterbank features, with models as small as 17k parameters.
Introduction
Wake-up word detection (WWD) serves as the primary interface for modern voice assistants. The task presents a unique engineering challenge: the system must remain active 24/7 to detect specific trigger phrases (e.g., "Hey Siri") while operating within the strict thermal and power constraints of microcontrollers or mobile DSPs.
Recent breakthroughs in Self-Supervised Learning (SSL) have demonstrated that models pre-trained on vast amounts of unlabeled audio can learn universal speech representations that generalize well across various tasks. However, these models—often comprising hundreds of millions of parameters—are far too large for real-time inference on the edge. Consequently, most commercial WWD systems still rely on hand-crafted features like log-Mel filterbanks (Fbank). This research explores whether the "intelligence" of a large SSL model can be distilled into a lightweight feature encoder that replaces traditional Fbank features without exceeding the resource limits of edge hardware.
Methodology
1. Architecture Pruning
The researchers observed that while the Transformer layers in models like wav2vec 2.0 provide essential global context for complex tasks like speech recognition, the CNN-based feature encoder alone captures sufficient local acoustic information for keyword detection. LiteFEW discards the Transformer layers entirely and further reduces the CNN encoder's size using a width multiplier ($\alpha$). This allows for a drastic reduction in parameters while maintaining the hierarchical feature extraction capabilities of the original model.
2. Auto-encoder Based Distillation
A significant challenge in knowledge distillation is the dimensionality mismatch between a large teacher model and a tiny student model. To bridge this gap, LiteFEW introduces an auto-encoder framework.
Instead of forcing the student to directly mimic the high-dimensional latent space of the teacher, an auto-encoder is trained on the teacher's representations. This auto-encoder learns to compress the teacher's knowledge into a lower-dimensional bottleneck. The student model is then trained to predict these compressed representations. This "translation" step ensures that the student focuses on the most salient features that its limited capacity can actually represent.
3. Two-Stage Optimization
The training process is divided into two distinct phases:
- Distillation Stage: The student encoder learns to reproduce the teacher's compressed speech representations using a combination of feature-based and response-based knowledge.
- Fine-tuning Stage: The distilled encoder is frozen, and a small WWD classifier (such as TC-ResNet or MatchboxNet) is trained on top. The researchers utilized focal loss during this stage to address the extreme class imbalance inherent in WWD datasets, where negative (background) audio vastly outweighs positive trigger utterances.
Experimental Setup
The proposed method was evaluated using the Hey Snips dataset, an open-source benchmark for wake-up word detection. The dataset contains approximately 11,000 positive utterances and 96 hours of negative audio from 1,800 different speakers. LiteFEW was compared against a standard baseline using 40-dimensional log-Mel filterbank features across several popular lightweight architectures, including TC-ResNet, DS-ResNet, and MatchboxNet.
Results and Discussion
Performance Gains
LiteFEW consistently outperformed the Fbank baseline across all tested architectures. The researchers reported a relative improvement of over 20% in the False Rejection Rate (FRR) at a fixed false alarm rate. This suggests that the distilled SSL features provide a more robust representation of speech than traditional hand-crafted features, especially in varied acoustic environments.
Extreme Efficiency
The efficiency of LiteFEW is highlighted by its "Small" configuration:
- Parameters: 17,000 (approximately 0.01% of the size of wav2vec 2.0).
- Performance: Even at this extreme scale, LiteFEW provided an 8.2% improvement over the Fbank baseline.
- Scalability: Increasing the parameter count to 64,000 (LiteFEW-Base) further boosted performance, demonstrating a clear scaling law between encoder capacity and detection accuracy.
Architecture Agnosticism
One of the most significant findings was that LiteFEW acts as a "plug-and-play" enhancement. Whether the underlying classifier was based on Dilated Convolutions or Depthwise Separable ResNets, the switch from Fbank to LiteFEW features resulted in a performance uplift without requiring changes to the classifier's internal structure.
Conclusion
LiteFEW demonstrates that the benefits of large-scale self-supervised speech models are not exclusive to high-power computing environments. By using an auto-encoder to distill and compress SSL representations, it is possible to create feature encoders that are both lightweight enough for microcontrollers and more accurate than traditional signal processing methods. This work paves the way for a new generation of highly efficient, SSL-powered voice interfaces on the edge.
Future research may explore the use of different SSL teachers, such as HuBERT or Data2Vec, and evaluate the framework's robustness against more aggressive, non-stationary industrial noise environments.
Summary updated