Introduction
While end-to-end (E2E) Automatic Speech Recognition (ASR) models like Transformers have achieved state-of-the-art accuracy, their deployment on resource-constrained smart devices remains a significant challenge. These models typically feature massive parameter counts, leading to high memory, power, and latency footprints that exceed the capabilities of wearables and smart speakers. This research by Alibaba and the Singapore Institute of Technology introduces a highly compact E2E ASR system that achieves over 10× model compression with minimal impact on recognition accuracy.
Methodology
The researchers developed a three-stage compression pipeline: cross-layer weight sharing, Taylor-expansion-based sparsification, and refined post-training quantization (PTQ).
Cross-layer Weight Sharing
To prevent parameter explosion in deep networks, the authors designed an architecture where sub-layers—specifically Memory Equipped Self-Attention (SAN-M) and Feed-Forward Networks (FFN)—share weights across different layers. This approach, inspired by the ALBERT model, allows for deeper networks without increasing the memory footprint.
Model Sparsification
Rather than simple magnitude-based pruning, the system employs a non-structural sparsification method based on a Taylor expansion importance metric. This metric identifies weights that contribute least to the loss function, which are then zeroed out during a fine-tuning phase. This ensures that the most task-relevant connections are preserved.
Post-training Quantization (PTQ)
The final stage involves converting 32-bit floating-point weights to 8-bit integers. The authors introduced three specific refinements to standard PTQ:
- Refined KL Algorithm: Utilizes a "real histogram" mapping values to [0, 255] to better capture statistics for non-ReLU activations.
- Improved ADMM: An iterative optimization process to determine optimal scaling factors for weight quantization.
- Label-free Automatic Mixed Precision (AMP): A safety mechanism that identifies "abnormal" layers sensitive to quantization noise by calculating the Mean Squared Error (MSE) between quantized and original outputs. These layers are kept in float32 to maintain accuracy without requiring labeled data.
Experimental Results
The system was evaluated on the AISHELL-1 and AISHELL-2 Mandarin speech benchmarks. On the 1,000-hour AISHELL-2 corpus, the proposed methods demonstrated exceptional efficiency.
As shown in the results, the baseline model was compressed from 248MB to just 24MB (a 10.3× reduction). Crucially, the Character Error Rate (CER) only increased from 6.49% to 6.92%, an absolute change of just 0.43%. Even with 50% sparsity, the model maintained a robust CER of 6.95%.
Discussion and Conclusion
The study highlights that weight sharing is a "free" efficiency gain, often allowing deeper models to outperform shallower ones with unique weights. Furthermore, the combination of Taylor-based metrics and label-free AMP provides a sophisticated way to navigate the trade-offs between model size and accuracy.
Future work will explore structural sparsification to improve hardware acceleration on standard processors and joint optimization of sparsification and quantization stages. This research provides a viable path for deploying high-performance E2E ASR on the next generation of smart devices.
Summary updated