Introduction
Automatic Speech Recognition (ASR) on edge devices, such as smartphones and smart speakers, requires a delicate balance between model accuracy and hardware constraints. Traditional models like the Recurrent Neural Network Transducer (RNN-T) apply a fixed computational cost to every audio frame. However, speech is inherently variable; periods of silence or simple phonemes require less processing power than dense, acoustically complex segments.
When fixed-size models encounter complex audio on low-power hardware, they often fall behind, creating a data backlog that leads to significant User-Perceived Latency (UPL). While existing adaptive compute methods target average computation reduction, they often fail to address real-time latency spikes. This paper introduces Amortized Neural Networks (AmNets), which dynamically switch between "fast" and "slow" processing branches to maintain real-time performance.
The Amortized RNN Architecture
The core of this approach is the Amortized RNN (AmRNN) layer. Unlike standard RNN cells, the AmRNN incorporates a dynamic execution path controlled by three primary components:
- The Arbitrator: A lightweight RNN that analyzes incoming audio frames to decide the optimal processing branch.
- Encoder Branches: Multiple versions of the layer with varying capacities. The authors explored Sparse Pruning (where one branch has 80% of weights zeroed) and Matrix Factorization (MF) (using SVD to create slimmed-down weight matrices).
- The Sampler/Combiner: This component manages branch selection. During training, it utilizes the Gumbel-Softmax trick for differentiability; at runtime, it functions as a hard switch for maximum efficiency.
Amortized Latency Loss ($L_{amr}$)
The primary innovation enabling real-time performance is the Amortized Latency Loss ($L_{amr}$). Standard loss functions typically penalize average FLOPs, which does not account for temporal backlogs. $L_{amr}$ mathematically models the "FLOP backlog" by rewarding the arbitrator for selecting the fast branch specifically when the processing buffer is near capacity. This ensures the model "catches up" during easier audio segments, keeping the system as close to real-time as possible. The loss function is hardware-aware, incorporating parameters for processor speed ($\mu$) and audio frame rate ($\rho$).
Experimental Results
The researchers evaluated AmNets using the LibriSpeech corpus (960 hours of English speech). To simulate real-world constraints, they tested the models in a resource-limited environment capped at 650 million FLOPs per second.
As shown in the results, the baseline model suffered from a massive latency of 6,154ms. By applying the Sparse AmNet with $L_{amr}$, latency was slashed to just 1.87ms—effectively real-time—while maintaining a Word Error Rate (WER) of 8.7%, compared to the baseline's 8.5%. The Matrix Factorization (MF) variant achieved a 45.6% reduction in FLOPs per frame. Crucially, models trained with $L_{amr}$ significantly outperformed those using standard average-compute loss ($L_{avg}$), proving that backlog awareness is the key to low-latency streaming.
Conclusion and Future Work
AmNets demonstrate that when computation is saved is more critical than how much is saved on average. By allowing models to dynamically allocate resources, AmNets achieve massive efficiency gains with negligible overhead (the arbitrator adds less than 2.5% to parameter counts).
Future research may explore multi-branch hierarchies for finer control and the application of AmNet principles to the decoder and joint networks of the RNN-T architecture. Additionally, further testing in high-noise or multi-speaker environments will be necessary to validate the arbitrator's robustness in diverse acoustic conditions.
Summary updated