TL;DR
This paper introduces a compressed Time Delay Neural Network (TDNN) for small-footprint keyword spotting, achieving significant improvements in detection accuracy while maintaining low computational requirements. By combining transfer learning, multi-task learning, and a novel application of Singular Value Decomposition (SVD) for compression, the authors demonstrate a more efficient and effective approach for embedded voice assistants.
What Problem Does This Paper Address?
The increasing prevalence of voice-activated assistants in devices like smart speakers and mobile phones necessitates keyword spotting systems that are not only accurate but also extremely efficient. These "small-footprint" systems must operate continuously on resource-constrained hardware, meaning they need to be lightweight in terms of CPU usage, memory footprint, and latency. Traditional approaches, often relying on large vocabulary continuous speech recognition (LVCSR) systems, are too computationally expensive. While existing small-footprint methods using DNNs or RNNs have shown promise, there's a continuous need to push the boundaries of efficiency without sacrificing detection performance, especially for high recall (to make devices easy to use) and low false accept rates (to ensure privacy).
Methods
The core of this work is a Time Delay Neural Network (TDNN) architecture adapted for keyword spotting. The authors employ several key techniques:
- TDNN Architecture: TDNNs are designed to process temporal information hierarchically. Initial layers capture narrow context, while deeper layers learn from wider temporal contexts. This is achieved through temporal connections that are "sub-sampled," reducing computational complexity by only computing activations at specific time intervals.
- Transfer Learning: To bootstrap the training process, the TDNN model is initialized with weights from a pre-trained LVCSR TDNN. This leverages knowledge gained from a related, larger-scale task.
- Multi-Task Learning: The TDNN is trained simultaneously on two tasks: keyword spotting (the primary task) and LVCSR phone targets (an auxiliary task). This helps regularize the training and improve the performance of the main keyword spotting task.
- Singular Value Decomposition (SVD) Compression: This is a novel contribution. The authors propose applying SVD to the weight matrices of the TDNN's affine transformations. This effectively introduces "bottleneck" layers, reducing the number of parameters and computational cost. Crucially, they first train a larger, full-rank TDNN and then compress it using SVD, allowing them to achieve better performance than training a small TDNN directly.
Datasets
The experiments were conducted on a large, in-house far-field corpus. This dataset is noted to contain significantly more instances of keyword and background speech utterances than previously used large-scale datasets for similar research. The authors do not introduce a new benchmark dataset but rather utilize their extensive internal data for training, tuning, and testing.
Key Insights and Ideas
- TDNNs Outperform DNNs: Even without compression, the TDNN architecture shows superior performance compared to a similarly sized DNN baseline for keyword spotting. This highlights the benefit of TDNNs' hierarchical temporal modeling.
- SVD Compression is Key for Efficiency and Performance: Applying SVD compression to TDNNs not only reduces model size but also acts as a regularizer, leading to improved accuracy. The strategy of training a larger model first and then compressing it via SVD proves more effective than directly training a compressed model.
- Synergy of Techniques: The combination of transfer learning, multi-task learning, temporal sub-sampling, and SVD compression creates a powerful pipeline for building highly efficient and accurate keyword spotting systems.
Key Results
The paper reports significant improvements in detection accuracy, measured by the Area Under the Curve (AUC) of the Detection Error Tradeoff (DET) curves.
- A TDNN without SVD compression (TDNN-A) achieved a 19.7% AUC reduction compared to a SVD-compressed DNN baseline.
- A TDNN with SVD compression (TDNN-B), trained using the proposed method, achieved a remarkable 37.6% AUC reduction compared to the same SVD-compressed DNN baseline.
- All models (DNN with SVD, TDNN without SVD, TDNN with SVD) were designed to have comparable numbers of parameters (under 100k) and similar total temporal context, ensuring a fair comparison.
Limitations and Future Work
- Limitations: The paper focuses on a specific keyword ("Alexa") and a particular type of acoustic environment (far-field). The absolute false accept rates are obscured due to confidentiality, making direct comparison with other published works on this metric challenging.
- Future Work: Not explicitly detailed, but the success of the TDNN-HMM approach with SVD compression suggests further exploration in optimizing the SVD compression strategy, exploring different TDNN architectures, and testing on a wider variety of keywords and acoustic conditions. The paper's focus on small-footprint systems also implies potential for further research into on-device training or adaptation.
Summary updated