Memory-efficient Speech Recognition on Smart Devices

Ganesh Venkatesh, Alagappan Valliappan, Jay Mahadeokar, Yuan Shangguan, Christian Fuegen, Michael L. Seltzer, Vikas Chandra

Abstract

Recurrent transducer models have emerged as a promising solution for speech recognition on the current and next generation smart devices. The transducer models provide competitive accuracy within a reasonable memory footprint alleviating the memory capacity constraints in these devices. However, these models access parameters from off-chip memory for every input time step which adversely effects device battery life and limits their usability on low-power devices. We address transducer model's memory access concerns by optimizing their model architecture and designing novel recurrent cell designs. We demonstrate that i) model's energy cost is dominated by accessing model weights from off-chip memory, ii) transducer model architecture is pivotal in determining the number of accesses to off-chip memory and just model size is not a good proxy, iii) our transducer model optimizations and novel recurrent cell reduces off-chip memory accesses by 4.5x and model size by 2x with minimal accuracy impact.

Introduction: The Energy Bottleneck in On-Device ASR

On-device Automatic Speech Recognition (ASR) is critical for ensuring privacy and low-latency interactions on wearable and smart home devices. While Recurrent Transducer (RNN-T) models are favored for their compact size, they face a significant efficiency hurdle: memory bandwidth.

In standard RNN-T implementations, the model fetches large weight matrices from off-chip memory (DRAM) for every 10ms audio frame. This process is energetically expensive; accessing off-chip memory consumes orders of magnitude more power than on-chip computation or local buffer (SRAM) access. Consequently, the true "cost" of a model on battery-constrained hardware is defined not just by parameter count, but by the frequency and volume of off-chip data transfers.

Energy cost breakdown showing that off-chip memory access dominates total energy consumption.

Architectural Innovations for Buffer-Aware Models

To address these constraints, we propose several architectural modifications designed to make the RNN-T encoder "buffer-aware." The goal is to ensure that active weights can reside within small on-chip buffers (e.g., 512KB) during execution.

Two-dimensional Cell Memory

Standard LSTM memory is represented as a vector. Reducing the hidden state size ($h$) to save parameters often compromises memory capacity. We introduce a 2D cell state ($h \times v$), where $v$ represents the number of memory vectors. This decoupling allows for a "skinny" network with low $h$ (reducing weight matrix size) that maintains high memory capacity through a larger $v$.

Internally Stacked Recurrent Cells and Scheduling

Instead of stacking multiple independent layers, we "stack" depth within the recurrent cell itself. This provides more non-linear processing power with a minimal increase in the parameter count. Furthermore, by optimizing scheduling—batching time steps to transition from matrix-vector to matrix-matrix operations—the model can move from being memory-bound (waiting for DRAM) to compute-bound, significantly improving energy efficiency.

Experimental Results

The proposed models were evaluated on the LibriSpeech corpus using the 960-hour training set. We compared our "Memory-Opt" configurations against a standard RNN-T baseline to measure the trade-offs between size, memory traffic, and accuracy.

Performance comparison of baseline (B) and memory-optimized models (E3, E7) across parameters, accuracy, and memory access.

As shown in the results, the optimized E7 model achieved a 4.5× reduction in off-chip memory traffic (dropping from 1.0× to 0.22×). This was accompanied by a reduction in encoder size from 32.8MB to 13.2MB (~2.5× reduction). Crucially, these efficiency gains were achieved with minimal impact on accuracy, with the Word Error Rate (WER) increasing only slightly from 4.63% to 4.87% in the most compressed version.

Conclusion

By redesigning recurrent cells to use 2D memory and optimizing layer scheduling for on-chip buffers, we demonstrate that RNN-T models can be made significantly more energy-efficient for smart devices. These findings suggest that model architecture, rather than just total size, is the primary lever for optimizing ASR on low-power hardware. Future work will explore combining these architectural changes with pruning and quantization to further push the boundaries of on-device efficiency.

Summary updated

Abstract

Recurrent transducer models have emerged as a promising solution for speech recognition on the current and next generation smart devices. The transducer models provide competitive accuracy within a reasonable memory footprint alleviating the memory capacity constraints in these devices. However, these models access parameters from off-chip memory for every input time step which adversely effects device battery life and limits their usability on low-power devices. We address transducer model's memory access concerns by optimizing their model architecture and designing novel recurrent cell designs. We demonstrate that i) model's energy cost is dominated by accessing model weights from off-chip memory, ii) transducer model architecture is pivotal in determining the number of accesses to off-chip memory and just model size is not a good proxy, iii) our transducer model optimizations and novel recurrent cell reduces off-chip memory accesses by 4.5x and model size by 2x with minimal accuracy impact.
BibTeX
@inproceedings{venkatesh2021memory,
  author = {Ganesh Venkatesh and Alagappan Valliappan and Jay Mahadeokar and Yuan Shangguan and Christian Fuegen and Michael L. Seltzer and Vikas Chandra},
  title = {{Memory-efficient Speech Recognition on Smart Devices}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=ebKRvuyT94w},
}
Figure