GhostRNN: Reducing State Redundancy in RNN with Cheap Operations

Hang Zhou, Xiaoxu Zheng, Yunhe Wang, Michael Bi Mi, Deyi Xiong, Kai HanarXiv 2024 · DOI · arXiv

Introduction

Recurrent Neural Networks (RNNs), particularly Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTM) networks, are foundational to real-time speech processing tasks such as keyword spotting and speech enhancement. However, their deployment on resource-constrained edge devices is often hindered by high memory requirements and computational complexity. While prior optimization efforts have targeted internal gating mechanisms, this research identifies a significant source of inefficiency in the hidden states themselves.

The Redundancy Problem

The authors observe that trained RNNs exhibit substantial redundancy across hidden units. Mathematical analysis using Singular Value Decomposition (SVD) reveals that approximately 50% of the singular values in a standard RNN hidden state map account for 99% of the information. This suggests that a large portion of the state vector is effectively redundant, performing overlapping representational work.

Cosine similarity matrix of hidden states, where red indicates high correlation and redundancy.

As shown in the similarity matrix above, many hidden units produce nearly identical patterns over time (correlation near 1.0), indicating that the model is wasting parameters and Multiply-Accumulate (MAC) operations to generate redundant features.

The GhostRNN Architecture

To address this, the authors propose GhostRNN, a module inspired by the "GhostNet" architecture from computer vision. GhostRNN replaces the standard heavy state generation with a two-stage process:

  1. Intrinsic State Generation: The model calculates a reduced set of "intrinsic states" (e.g., 50% of the total desired dimension) using standard RNN/GRU logic.
  2. Ghost State Generation: The remaining "ghost states" are derived from the intrinsic states using "cheap operations"—specifically, simple linear transformations followed by an activation function.
  3. Concatenation: The intrinsic and ghost states are merged to form the final hidden state for the next time step.
Comparison between a standard RNN and the proposed GhostRNN architecture.

By utilizing cheap linear layers instead of full matrix multiplications for a significant portion of the state vector, GhostRNN maintains high representational capacity while drastically reducing the parameter count and computational load.

Experimental Evaluation

The GhostRNN framework was evaluated on two critical speech tasks:

Keyword Spotting (KWS)

Using the Google Speech Commands v0.02 dataset, GhostRNN achieved 94.79% accuracy with only 292K parameters. This outperforms a standard GRU baseline, which required 498K parameters to achieve a lower accuracy of 94.68%.

Speech Enhancement (SE)

On the LibriMix dataset, GhostRNN demonstrated significant efficiency gains across different model backbones:

  • DCCRN: Reduced parameters from 3.4M to 3.1M while maintaining Signal-to-Distortion Ratio (SDR) performance.
  • GRU-TasNet: Reduced parameters by approximately 40% (from 5.6M to 3.4M) while actually improving the SDR from 13.14 dB to 13.26 dB.

Conclusion

GhostRNN provides a robust method for compressing RNNs by exploiting state redundancy. By generating a fraction of the hidden states through computationally inexpensive linear operations, the architecture achieves roughly a 40% reduction in parameters and MACs without performance degradation. This makes high-quality speech processing more accessible for low-power, real-time edge applications.

Summary updated

Abstract

Recurrent neural network (RNNs) that are capable of modeling long-distance dependencies are widely used in various speech tasks, eg., keyword spotting (KWS) and speech enhancement (SE). Due to the limitation of power and memory in low-resource devices, efficient RNN models are urgently required for real-world applications. In this paper, we propose an efficient RNN architecture, GhostRNN, which reduces hidden state redundancy with cheap operations. In particular, we observe that partial dimensions of hidden states are similar to the others in trained RNN models, suggesting that redundancy exists in specific RNNs. To reduce the redundancy and hence computational cost, we propose to first generate a few intrinsic states, and then apply cheap operations to produce ghost states based on the intrinsic states. Experiments on KWS and SE tasks demonstrate that the proposed GhostRNN significantly reduces the memory usage (~40%) and computation cost while keeping performance similar.
BibTeX
@misc{zhou2024ghostrnn,
  author = {Hang Zhou and Xiaoxu Zheng and Yunhe Wang and Michael Bi Mi and Deyi Xiong and Kai Han},
  title = {{GhostRNN: Reducing State Redundancy in RNN with Cheap Operations}},
  year = {2024},
  eprint = {2411.14489},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.14489},
}
Figure