Conformer-Based Speech Recognition On Extreme Edge-Computing Devices

Mingbin Xu, Alex Jin, Sicheng Wang, Mu Su, Tim Ng, Henry Mason, Shiyi Han, Zhihong Lei, Yaqiao Deng, Zhen Huang, Mahesh KrishnamoorthyNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of running powerful Conformer-based speech recognition models on resource-constrained edge devices like smartphones and wearables. They achieve this through a suite of optimizations, including architectural changes, graph transformations, and numerical stability techniques, resulting in significantly faster inference with minimal accuracy loss and reduced energy consumption.

What Problem Does This Paper Address?

The increasing power of edge devices has enabled a shift of computationally intensive tasks, like Automatic Speech Recognition (ASR), from the cloud to the device itself. This move offers benefits like enhanced user privacy and reduced latency. However, implementing advanced ASR models, particularly state-of-the-art Conformer architectures, on devices with limited processing power, memory, and energy budgets remains a significant hurdle. These devices often lack the robust computational resources found in servers, leading to challenges in achieving real-time performance without sacrificing accuracy or draining batteries.

Methods

The authors propose a multi-pronged approach to optimize Conformer-based ASR for edge devices:

  • Depthwise Separable Convolutions (DWS): They replace the standard convolution layers in the Conformer's subsampling module with DWS. This significantly reduces computational cost (from 32.8% to 4.0% of overall computation) while maintaining ASR accuracy.
  • Memory-Aware Graph Execution: Inspired by Apple's principles for optimizing models on their Neural Engine (ANE), they focus on efficient data handling. This includes:
  • Data Format Optimization: Aligning tensor representations with the ANE's architecture.
  • Tensor Chunking: Splitting large intermediate tensors to improve L2 cache utilization.
  • Minimizing Memory Copies: Replacing operations like reshape and transpose with more efficient alternatives, such as Einstein summation for batch matrix multiplications.
  • Handling Bandwidth-Boundness: Carefully benchmarking and making informed decisions about memory operations.
  • Stability of Layer Normalization: A key contribution is a theoretical framework and practical solution for stabilizing Layer Normalization (Layernorm) in low-precision environments common on edge hardware. They introduce a "Mean Absolute Deviation (MAD) normalization" as a pre-normalizer. This technique, derived from theoretical analysis of Lp-norms, maps input distributions to a bounded region, preventing numerical underflows/overflows during Layernorm computation without requiring model retraining.
  • Scaling of Softmax: To address hardware accelerators' potential lack of support for exponential operations, they propose a conditional re-scaling technique for softmax layers. This method aims to preserve accuracy while enabling efficient computation on hardware with limited capabilities.

Datasets

The paper mentions using a training corpus of 17k hours of anonymized virtual assistant queries. For evaluation, they created a test set of 20k queries, with 200 queries sampled for performance testing.

Key Insights and Ideas

  • DWS is a win-win: Depthwise separable convolutions offer a substantial computational advantage for subsampling in Conformer models without a significant hit to accuracy, making them ideal for edge deployment.
  • Low-precision stability is crucial: Modern hardware accelerators rely on low-precision arithmetic for efficiency, but this can lead to numerical instability in neural network layers like Layernorm. The paper provides a theoretical and practical solution to this problem.
  • Pre-training Layernorm matters: Introducing a pre-normalizer before Layernorm can prevent numerical issues in low-precision settings, and this technique is generalizable beyond just ASR.
  • Hardware-aware optimization is key: Understanding the specific constraints and capabilities of target hardware (like Apple's ANE) is essential for effective model deployment, guiding choices in data formats, memory access, and operation implementation.
  • Softmax can be tricky on edge: Hardware limitations might necessitate approximations or alternative implementations for operations like softmax, and the proposed conditional re-scaling offers a way to manage this.

Key Results

  • Speedup: Achieved over 5.26 times faster than real-time (0.19 RTF) on small wearables.
  • Energy Reduction: Reported a reduction in energy consumption by an order of magnitude when using hardware accelerators.
  • Accuracy: Maintained state-of-the-art accuracy, with negligible differences between FP16 and FP32 precision, and DWS models performing comparably to standard convolution models.
  • Computational Reduction: DWS subsampling reduced computational bottleneck from 32.8% to 4.0%.
  • Overflow Reduction: The proposed Layernorm stabilization technique significantly reduced overflow statistics in low-precision computations.

Limitations and Future Work

  • Softmax WER Regression: The proposed softmax modification led to a slight WER regression, though it was deemed acceptable for low WER scenarios.
  • Generalizability of MAD: While theoretically sound and empirically validated for common distributions, the paper notes that the normalization constant used in their experiments was slightly lower than theoretically optimal, as extreme value distributions are rare in practice (e.g., Gaussian distributions).
  • Future Work: The authors suggest their theoretical findings on numerical stabilization are applicable to a wide range of deep learning models and computing tasks, implying broader potential applications. The paper also mentions that the encoder's workload dominates on watches, suggesting further optimization efforts could focus there.

Summary updated

BibTeX
@inproceedings{xu2024conformer,
  author = {Mingbin Xu and Alex Jin and Sicheng Wang and Mu Su and Tim Ng and Henry Mason and Shiyi Han and Zhihong Lei and Yaqiao Deng and Zhen Huang and Mahesh Krishnamoorthy},
  title = {{Conformer-Based Speech Recognition On Extreme Edge-Computing Devices}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-industry.12},
}
Figure