Abstract
Deploying high-performance speaker recognition on ultra-low-power mobile hardware requires a delicate balance between computational efficiency and accuracy. This paper explores the application of structural sparsification to Time-Delay Neural Networks (TDNNs) for far-field speaker recognition. By utilizing Group Lasso penalties to enforce hardware-friendly sparsity, we achieve a 60% reduction in model parameters and a 1.5× execution speedup on the Intel® GNA accelerator with negligible impact on accuracy.
Introduction
Far-field speaker recognition is a critical component for smart devices, yet its reliance on deep neural networks (DNNs) often exceeds the power and computational budgets of mobile processors. While traditional weight pruning can reduce model size, it typically results in unstructured sparsity that standard hardware cannot exploit for speedups. To address this, we propose a structural sparsification approach that removes entire blocks or "chunks" of weights, aligning with the processing widths of specialized hardware like the Intel® GNA.
Methodology
Our approach optimizes the popular x-vector model architecture through three primary modifications:
- Topology Simplification: The standard x-vector model was streamlined into a 5-layer CNN, treating the TDNN as a 1D convolution. Embedding dimensions were reduced to establish a lower-complexity baseline.
- End-to-End Scoring: We employed Additive Margin Softmax (AM-Softmax) during training. This enables the use of simple cosine distance for scoring during inference, bypassing the need for computationally expensive back-ends like PLDA and allowing the entire pipeline to run on the hardware accelerator.
- Structural Sparsification via Group Lasso: To ensure hardware compatibility, we introduced a Group Lasso penalty to the loss function. This penalty encourages entire "chunks" of weights (specifically 8 or 16 elements, matching the Intel GNA’s processing width) to drop to zero simultaneously.
Experimental Setup
Models were trained on the VoxCeleb 1 and 2 datasets, encompassing over 7,000 identities. To simulate far-field conditions, we applied 9× data augmentation using room impulse responses (RIRs) and background noise from the MUSAN and AudioSet corpuses. Evaluation was conducted on the VOiCES far-field dataset, which mimics challenging acoustic environments characterized by significant reverberation and noise.
Key Results
The proposed structural sparsification significantly improved efficiency without sacrificing performance. The method successfully removed 60% of parameters, while the Equal Error Rate (EER) only saw a marginal increase from 1.81% to 1.99%.
On an Intel Pentium Silver J4005 using the GNA accelerator, the structural sparse model achieved over a 1.5× speedup in execution time. Notably, as shown in the performance curves, sparse models consistently outperformed dense models of equivalent parameter counts across all tested granularities.
Analysis of Sparsity and Granularity
Our analysis revealed several critical insights into the nature of model sparsity:
- Granularity and Performance: Smaller granularity (Chunk-8) achieved a lower EER than larger granularity (Filters) for the same parameter count. This suggests that finer-grained structural pruning preserves more representative features.
- Sparsity as a Regularizer: Sparse models occasionally outperformed dense models of the same size, indicating that the Group Lasso penalty acts as an effective regularizer, improving generalization to noisy, unseen data.
- Layer Sensitivity: Sparsity was most effective in the first four layers. The layers immediately preceding the statistics pooling layer were resistant to pruning, suggesting they contain the highest density of information required for speaker identification.
Conclusion
This study demonstrates that structural sparsification is a highly effective technique for deploying TDNN-based speaker recognition on low-power hardware. By aligning model architecture with hardware processing widths, we achieved significant speedups and size reductions with minimal accuracy loss. Future work will investigate applying these structural pruning techniques to more complex architectures, such as Transformers and Conformers, to further push the boundaries of mobile speech technology.
Summary updated