Extremely Low Footprint End-to-End ASR System for Smart Device

Zhifu Gao, Yiwu Yao, Shiliang Zhang, Jun Yang, Ming Lei, Ian McLoughlin

Introduction

While end-to-end (E2E) Automatic Speech Recognition (ASR) models like Transformers have achieved state-of-the-art accuracy, their deployment on resource-constrained smart devices remains a significant challenge. These models typically feature massive parameter counts, leading to high memory, power, and latency footprints that exceed the capabilities of wearables and smart speakers. This research by Alibaba and the Singapore Institute of Technology introduces a highly compact E2E ASR system that achieves over 10× model compression with minimal impact on recognition accuracy.

Methodology

The researchers developed a three-stage compression pipeline: cross-layer weight sharing, Taylor-expansion-based sparsification, and refined post-training quantization (PTQ).

Cross-layer Weight Sharing

To prevent parameter explosion in deep networks, the authors designed an architecture where sub-layers—specifically Memory Equipped Self-Attention (SAN-M) and Feed-Forward Networks (FFN)—share weights across different layers. This approach, inspired by the ALBERT model, allows for deeper networks without increasing the memory footprint.

Proposed encoder-decoder architecture with cross-layer weight sharing.

Model Sparsification

Rather than simple magnitude-based pruning, the system employs a non-structural sparsification method based on a Taylor expansion importance metric. This metric identifies weights that contribute least to the loss function, which are then zeroed out during a fine-tuning phase. This ensures that the most task-relevant connections are preserved.

Post-training Quantization (PTQ)

The final stage involves converting 32-bit floating-point weights to 8-bit integers. The authors introduced three specific refinements to standard PTQ:

  1. Refined KL Algorithm: Utilizes a "real histogram" mapping values to [0, 255] to better capture statistics for non-ReLU activations.
  2. Improved ADMM: An iterative optimization process to determine optimal scaling factors for weight quantization.
  3. Label-free Automatic Mixed Precision (AMP): A safety mechanism that identifies "abnormal" layers sensitive to quantization noise by calculating the Mean Squared Error (MSE) between quantized and original outputs. These layers are kept in float32 to maintain accuracy without requiring labeled data.

Experimental Results

The system was evaluated on the AISHELL-1 and AISHELL-2 Mandarin speech benchmarks. On the 1,000-hour AISHELL-2 corpus, the proposed methods demonstrated exceptional efficiency.

Performance comparison across different compression stages on AISHELL-2.

As shown in the results, the baseline model was compressed from 248MB to just 24MB (a 10.3× reduction). Crucially, the Character Error Rate (CER) only increased from 6.49% to 6.92%, an absolute change of just 0.43%. Even with 50% sparsity, the model maintained a robust CER of 6.95%.

Discussion and Conclusion

The study highlights that weight sharing is a "free" efficiency gain, often allowing deeper models to outperform shallower ones with unique weights. Furthermore, the combination of Taylor-based metrics and label-free AMP provides a sophisticated way to navigate the trade-offs between model size and accuracy.

Future work will explore structural sparsification to improve hardware acceleration on standard processors and joint optimization of sparsification and quantization stages. This research provides a viable path for deploying high-performance E2E ASR on the next generation of smart devices.

Summary updated

BibTeX
@inproceedings{gao2021extremely,
  author = {Zhifu Gao and Yiwu Yao and Shiliang Zhang and Jun Yang and Ming Lei and Ian McLoughlin},
  title = {{Extremely Low Footprint End-to-End ASR System for Smart Device}},
  booktitle = {Interspeech},
  year = {2021},
  url = {https://openreview.net/forum?id=Lq3WptNXE1},
}
Figure