Introduction
Standard Recurrent Neural Networks (RNNs) are powerful for sequence tasks like speech recognition, but they suffer from the vanishing gradient problem. When training with backpropagation through time, gradients tend to shrink as they move further back, making it nearly impossible for the network to learn long-term dependencies.
The industry-standard solution is the Long Short-Term Memory (LSTM) network. While effective, LSTMs are computationally expensive; an LSTM layer typically has four times as many parameters as a standard RNN layer of the same size due to its complex gating mechanisms. This research introduces High Order Recurrent Neural Networks (HORNNs) as a lightweight, efficient alternative that maintains long-term memory without the massive overhead.
Methodology
The authors propose the High Order RNN (HORNN). Unlike a standard RNN that only looks at the immediately preceding state ($h_{t-1}$), a HORNN incorporates states from further back in time ($h_{t-n}$).
Key technical innovations include:
- High Order Connections: The hidden state calculation is modified to include a "memory" term from $n$ steps ago. For ReLU-based networks, the formula is: $h_t = f(W x_t + U_1 h_{t-1} + U_n h_{t-n} + b)$.
- Sigmoid-Specific Architecture: For sigmoid activation functions, which are more prone to vanishing gradients, an extra direct residual-like connection ($h_{t-m}$) is added to the input of the activation function to ease gradient flow.
- Matrix Factorization (HORNNP): To keep parameter counts low, a linear projection layer factorizes large weight matrices into smaller, low-rank approximations (similar to LSTMP).
- p-sigmoid: A parameterized sigmoid with a trainable scaling factor is used to mimic ReLU behavior while maintaining sigmoid properties.
Experimental Setup
Models were evaluated on the MGB3 (Multi-Genre Broadcast) challenge dataset, consisting of British English television audio.
- Training: A 55-hour subset was used for architecture search, and the full 275-hour set was used for scaling validation.
- Evaluation: The dev17b set (5.55 hours) was used for testing Word Error Rate (WER).
- Language Model: A 63k word vocabulary with a trigram (tg) word-level language model and a confidence network (cn).
Key Results
HORNNs significantly outperform standard RNNs and match LSTM performance with much lower complexity.
- Performance vs. RNNs: HORNNs reduced WER by 4.2% (ReLU) and 6.3% (Sigmoid) compared to standard RNNs. The "High Order" connections act as shortcuts for gradients, allowing them to "jump" across time steps.
- Efficiency vs. LSTMs: HORNNs achieved similar WER to LSTMs while using only 20%–50% of the parameters. In the authors' implementation, this resulted in a 50% speedup in computation.
- Scaling: The benefits held true when scaling to the full 275-hour dataset.
As shown in the table above, the 2-layer ReLU HORNNP ($R_4^{275h}$) achieves a WER of 25.3 (tg), slightly better than the 2-layer LSTMP ($L_3^{275h}$) at 25.7 (tg), despite the HORNN's simpler architecture.
Conclusion and Future Work
HORNNs provide a compelling alternative to LSTMs by relaxing the first-order Markov property and using simpler gradient shortcuts instead of complex gating. This allows for more efficient use of computational resources. Future work could explore extremely deep architectures and the performance of HORNNs on other sequence tasks like machine translation or language modelling.
Summary updated