Abstract
Despite showing state-of-the-art performance, deep learning for speech recognition remains challenging to deploy in on-device edge scenarios such as mobile and other consumer devices. Recently, there have been greater efforts in the design of small, low-footprint deep neural networks (DNNs) that are more appropriate for edge devices, with much of the focus on design principles for hand-crafting efficient network architectures. In this study, we explore a human-machine collaborative design strategy for building low-footprint DNN architectures for speech recognition through a marriage of human-driven principled network design prototyping and machine-driven design exploration. The efficacy of this design strategy is demonstrated through the design of a family of highly-efficient DNNs (nicknamed EdgeSpeechNets) for limited-vocabulary speech recognition. Experimental results using the Google Speech Commands dataset for limited-vocabulary speech recognition showed that EdgeSpeechNets have higher accuracies than state-of-the-art DNNs (with the best EdgeSpeechNet achieving ~97% accuracy), while achieving significantly smaller network sizes (as much as 7.8x smaller) and lower computational cost (as much as 36x fewer multiply-add operations, 10x lower prediction latency, and 16x smaller memory footprint on a Motorola Moto E phone), making them very well-suited for on-device edge voice interface applications.
1. Introduction
Deep learning has revolutionized speech recognition, yet deploying these models on "the edge"—such as smart mirrors, IoT sensors, or budget smartphones—remains a significant challenge. These devices operate under strict constraints regarding memory, processing power, and battery life. A critical task for such devices is limited-vocabulary speech recognition, or keyword spotting, which enables devices to respond to specific verbal cues locally without requiring a constant cloud connection.
While manual "hand-crafting" of small networks has provided some relief, human designers often struggle to navigate the vast, high-dimensional space of possible network configurations. This paper addresses this "design gap" by proposing a collaborative approach that leverages machine-driven optimization to handle fine-grained architectural decisions.
2. Methodology: Human-Machine Collaborative Design
The researchers developed a two-stage human-machine collaborative design strategy to create the EdgeSpeechNet family.
2.1 Human-driven Design Prototyping
In the first stage, humans establish the foundational "principled" design. This includes selecting Mel-frequency cepstrum coefficient (MFCC) representations as the input, which transforms audio into a 2D frequency-over-time image. The backbone of the network is constructed using residual blocks (skip connections) to facilitate effective learning in deeper architectures.
2.2 Machine-driven Design Exploration
The second stage utilizes "generative synthesis" to explore the design space defined by the prototype. An algorithm searches for the optimal combination of network width, depth, and connectivity to maximize a "NetScore"—a universal performance function balancing accuracy, architectural complexity, and computational cost. This process ensures the final models meet a minimum accuracy threshold (95%) while being as efficient as possible.
3. EdgeSpeechNet Architectures
This collaborative process resulted in four distinct models: EdgeSpeechNet-A, B, C, and D. Unlike traditional manual designs that often use uniform layer widths, the machine-driven exploration produced fine-grained, non-uniform architectures tailored for efficiency. These models represent different trade-offs on the accuracy-vs-efficiency spectrum.
4. Experimental Results
The EdgeSpeechNets were evaluated using the Google Speech Commands dataset, consisting of 65,000 one-second audio clips of 30 short words. The models consistently outperformed existing state-of-the-art architectures like res15 and Google’s tpool2.
- Accuracy: EdgeSpeechNet-A achieved approximately 97% accuracy, surpassing the much larger
res15model. - Efficiency: EdgeSpeechNet-D demonstrated extreme efficiency, requiring 36.5x fewer multiply-add operations and maintaining a 7.8x smaller footprint than
res15. - On-Device Performance: When deployed on a budget Motorola Moto E phone, EdgeSpeechNet-D achieved a latency of just 34ms and a memory footprint of ~1MB. This represents a 10x speed improvement and a 16x reduction in memory usage compared to previous benchmarks.
5. Conclusion
The EdgeSpeechNet family demonstrates that a "marriage" of human macro-design and machine micro-design can produce neural networks that are both more accurate and significantly more efficient than those designed by humans alone. While currently focused on limited-vocabulary tasks, this collaborative design strategy holds promise for broader applications in continuous speech-to-text, visual perception, and other natural language processing tasks where edge deployment is a critical bottleneck.
Summary updated