Introduction
Recurrent Neural Networks (RNNs), particularly Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) networks, are foundational to real-time speech processing tasks such as keyword spotting and speech enhancement. However, their deployment on resource-constrained edge devices is often hindered by high memory requirements and computational complexity. While prior optimization efforts have targeted internal gating mechanisms, this research identifies a significant source of inefficiency in the hidden states themselves.
The Redundancy Problem
The authors observe that trained RNNs exhibit substantial redundancy across hidden units. Mathematical analysis using Singular Value Decomposition (SVD) reveals that approximately 50% of the singular values in a standard RNN hidden state map account for 99% of the information. This suggests that a large portion of the state vector is effectively redundant, performing overlapping representational work.
As shown in the similarity matrix above, many hidden units produce nearly identical patterns over time (correlation near 1.0), indicating that the model is wasting parameters and Multiply-Accumulate (MAC) operations to generate redundant features.
The GhostRNN Architecture
To address this, the authors propose GhostRNN, a module inspired by the "GhostNet" architecture from computer vision. GhostRNN replaces the standard heavy state generation with a two-stage process:
- Intrinsic State Generation: The model calculates a reduced set of "intrinsic states" (e.g., 50% of the total desired dimension) using standard RNN/GRU logic.
- Ghost State Generation: The remaining "ghost states" are derived from the intrinsic states using "cheap operations"—specifically, simple linear transformations followed by an activation function.
- Concatenation: The intrinsic and ghost states are merged to form the final hidden state for the next time step.
By utilizing cheap linear layers instead of full matrix multiplications for a significant portion of the state vector, GhostRNN maintains high representational capacity while drastically reducing the parameter count and computational load.
Experimental Evaluation
The GhostRNN framework was evaluated on two critical speech tasks:
Keyword Spotting (KWS)
Using the Google Speech Commands v0.02 dataset, GhostRNN achieved 94.79% accuracy with only 292K parameters. This outperforms a standard GRU baseline, which required 498K parameters to achieve a lower accuracy of 94.68%.
Speech Enhancement (SE)
On the LibriMix dataset, GhostRNN demonstrated significant efficiency gains across different model backbones:
- DCCRN: Reduced parameters from 3.4M to 3.1M while maintaining Signal-to-Distortion Ratio (SDR) performance.
- GRU-TasNet: Reduced parameters by approximately 40% (from 5.6M to 3.4M) while actually improving the SDR from 13.14 dB to 13.26 dB.
Conclusion
GhostRNN provides a robust method for compressing RNNs by exploiting state redundancy. By generating a fraction of the hidden states through computationally inexpensive linear operations, the architecture achieves roughly a 40% reduction in parameters and MACs without performance degradation. This makes high-quality speech processing more accessible for low-power, real-time edge applications.
Summary updated