EdgeSpeechNets: Highly Efficient Deep Neural Networks for Speech Recognition on the Edge

Zhong Qiu Lin, Audrey G. Chung, Alexander WongarXiv 2018 · OpenReview

Abstract

Despite showing state-of-the-art performance, deep learning for speech recognition remains challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Recently, there have been greater efforts in the design of small, low-footprint deep neural networks (DNNs) that are more appropriate for edge devices, with much of the focus on design principles for hand-crafting efficient network architectures. In this study, we explore a human-machine collaborative design strategy for building low-footprint DNN architectures for speech recognition through a marriage of human-driven principled network design prototyping and machine-driven design exploration. The efficacy of this design strategy is demonstrated through the design of a family of highly-efficient DNNs (nicknamed EdgeSpeechNets) for limited-vocabulary speech recognition. Experimental results using the Google Speech Commands dataset for limited-vocabulary speech recognition showed that EdgeSpeechNets have higher accuracies than state-of-the-art DNNs (with the best EdgeSpeechNet achieving ~97% accuracy), while achieving significantly smaller network sizes (as much as 7.8x smaller) and lower computational cost (as much as 36x fewer multiply-add operations, 10x lower prediction latency, and 16x smaller memory footprint on a Motorola Moto E phone), making them very well-suited for on-device edge voice interface applications.

1. Introduction

Deep learning has revolutionized speech recognition, yet deploying these models on "the edge"—such as smart mirrors, IoT sensors, or budget smartphones—remains a significant challenge. These devices operate under strict constraints regarding memory, processing power, and battery life. A critical task for such devices is limited-vocabulary speech recognition, or keyword spotting, which enables devices to respond to specific verbal cues locally without requiring a constant cloud connection.

While manual "hand-crafting" of small networks has provided some relief, human designers often struggle to navigate the vast, high-dimensional space of possible network configurations. This paper addresses this "design gap" by proposing a collaborative approach that leverages machine-driven optimization to handle fine-grained architectural decisions.

2. Methodology: Human-Machine Collaborative Design

The researchers developed a two-stage human-machine collaborative design strategy to create the EdgeSpeechNet family.

2.1 Human-driven Design Prototyping

In the first stage, humans establish the foundational "principled" design. This includes selecting Mel-frequency cepstrum coefficient (MFCC) representations as the input, which transforms audio into a 2D frequency-over-time image. The backbone of the network is constructed using residual blocks (skip connections) to facilitate effective learning in deeper architectures.

The EdgeSpeechNet design pipeline, illustrating the transformation from audio signal to MFCCs and the subsequent residual-based neural architecture.

2.2 Machine-driven Design Exploration

The second stage utilizes "generative synthesis" to explore the design space defined by the prototype. An algorithm searches for the optimal combination of network width, depth, and connectivity to maximize a "NetScore"—a universal performance function balancing accuracy, architectural complexity, and computational cost. This process ensures the final models meet a minimum accuracy threshold (95%) while being as efficient as possible.

3. EdgeSpeechNet Architectures

This collaborative process resulted in four distinct models: EdgeSpeechNet-A, B, C, and D. Unlike traditional manual designs that often use uniform layer widths, the machine-driven exploration produced fine-grained, non-uniform architectures tailored for efficiency. These models represent different trade-offs on the accuracy-vs-efficiency spectrum.

Detailed architectural parameters for two EdgeSpeechNet variants, illustrating the fine-grained, non-uniform layer configurations discovered via generative synthesis.

4. Experimental Results

The EdgeSpeechNets were evaluated using the Google Speech Commands dataset, consisting of 65,000 one-second audio clips of 30 short words. The models consistently outperformed existing state-of-the-art architectures like res15 and Google’s tpool2.

  • Accuracy: EdgeSpeechNet-A achieved approximately 97% accuracy, surpassing the much larger res15 model.
  • Efficiency: EdgeSpeechNet-D demonstrated extreme efficiency, requiring 36.5x fewer multiply-add operations and maintaining a 7.8x smaller footprint than res15.
  • On-Device Performance: When deployed on a budget Motorola Moto E phone, EdgeSpeechNet-D achieved a latency of just 34ms and a memory footprint of ~1MB. This represents a 10x speed improvement and a 16x reduction in memory usage compared to previous benchmarks.

5. Conclusion

The EdgeSpeechNet family demonstrates that a "marriage" of human macro-design and machine micro-design can produce neural networks that are both more accurate and significantly more efficient than those designed by humans alone. While currently focused on limited-vocabulary tasks, this collaborative design strategy holds promise for broader applications in continuous speech-to-text, visual perception, and other natural language processing tasks where edge deployment is a critical bottleneck.

Summary updated

Abstract

Despite showing state-of-the-art performance, deep learning for speech recognition remains challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Recently, there have been greater efforts in the design of small, low-footprint deep neural networks (DNNs) that are more appropriate for edge devices, with much of the focus on design principles for hand-crafting efficient network architectures. In this study, we explore a human-machine collaborative design strategy for building low-footprint DNN architectures for speech recognition through a marriage of human-driven principled network design prototyping and machine-driven design exploration. The efficacy of this design strategy is demonstrated through the design of a family of highly-efficient DNNs (nicknamed EdgeSpeechNets) for limited-vocabulary speech recognition. Experimental results using the Google Speech Commands dataset for limited-vocabulary speech recognition showed that EdgeSpeechNets have higher accuracies than state-of-the-art DNNs (with the best EdgeSpeechNet achieving ~97% accuracy), while achieving significantly smaller network sizes (as much as 7.8x smaller) and lower computational cost (as much as 36x fewer multiply-add operations, 10x lower prediction latency, and 16x smaller memory footprint on a Motorola Moto E phone), making them very well-suited for on-device edge voice interface applications.
BibTeX
@misc{lin2018edgespeechnets,
  author = {Zhong Qiu Lin and Audrey G. Chung and Alexander Wong},
  title = {{EdgeSpeechNets: Highly Efficient Deep Neural Networks for Speech Recognition on the Edge}},
  year = {2018},
  eprint = {1810.08559},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.08559},
}
Figure