High Order Recurrent Neural Networks for Acoustic Modelling

Chao Zhang, Philip C. Woodland

Introduction

Standard Recurrent Neural Networks (RNNs) are powerful for sequence tasks like speech recognition, but they suffer from the vanishing gradient problem. When training with backpropagation through time, gradients tend to shrink as they move further back, making it nearly impossible for the network to learn long-term dependencies.

The industry-standard solution is the Long Short-Term Memory (LSTM) network. While effective, LSTMs are computationally expensive; an LSTM layer typically has four times as many parameters as a standard RNN layer of the same size due to its complex gating mechanisms. This research introduces High Order Recurrent Neural Networks (HORNNs) as a lightweight, efficient alternative that maintains long-term memory without the massive overhead.

Methodology

The authors propose the High Order RNN (HORNN). Unlike a standard RNN that only looks at the immediately preceding state ($h_{t-1}$), a HORNN incorporates states from further back in time ($h_{t-n}$).

Key technical innovations include:

  • High Order Connections: The hidden state calculation is modified to include a "memory" term from $n$ steps ago. For ReLU-based networks, the formula is: $h_t = f(W x_t + U_1 h_{t-1} + U_n h_{t-n} + b)$.
  • Sigmoid-Specific Architecture: For sigmoid activation functions, which are more prone to vanishing gradients, an extra direct residual-like connection ($h_{t-m}$) is added to the input of the activation function to ease gradient flow.
  • Matrix Factorization (HORNNP): To keep parameter counts low, a linear projection layer factorizes large weight matrices into smaller, low-rank approximations (similar to LSTMP).
  • p-sigmoid: A parameterized sigmoid with a trainable scaling factor is used to mimic ReLU behavior while maintaining sigmoid properties.

Experimental Setup

Models were evaluated on the MGB3 (Multi-Genre Broadcast) challenge dataset, consisting of British English television audio.

  • Training: A 55-hour subset was used for architecture search, and the full 275-hour set was used for scaling validation.
  • Evaluation: The dev17b set (5.55 hours) was used for testing Word Error Rate (WER).
  • Language Model: A 63k word vocabulary with a trigram (tg) word-level language model and a confidence network (cn).

Key Results

HORNNs significantly outperform standard RNNs and match LSTM performance with much lower complexity.

Comparison of Baseline systems (top) and HORNN systems (bottom) across various hyperparameter settings (n, m).
  • Performance vs. RNNs: HORNNs reduced WER by 4.2% (ReLU) and 6.3% (Sigmoid) compared to standard RNNs. The "High Order" connections act as shortcuts for gradients, allowing them to "jump" across time steps.
  • Efficiency vs. LSTMs: HORNNs achieved similar WER to LSTMs while using only 20%–50% of the parameters. In the authors' implementation, this resulted in a 50% speedup in computation.
  • Scaling: The benefits held true when scaling to the full 275-hour dataset.
Final performance comparison on the 275-hour MGB3 dataset showing HORNNP matching or exceeding LSTMP performance.

As shown in the table above, the 2-layer ReLU HORNNP ($R_4^{275h}$) achieves a WER of 25.3 (tg), slightly better than the 2-layer LSTMP ($L_3^{275h}$) at 25.7 (tg), despite the HORNN's simpler architecture.

Conclusion and Future Work

HORNNs provide a compelling alternative to LSTMs by relaxing the first-order Markov property and using simpler gradient shortcuts instead of complex gating. This allows for more efficient use of computational resources. Future work could explore extremely deep architectures and the performance of HORNNs on other sequence tasks like machine translation or language modelling.

Summary updated

Abstract

Vanishing long-term gradients are a major issue in training standard recurrent neural networks (RNNs), which can be alleviated by long short-term memory (LSTM) models with memory cells. However, the extra parameters associated with the memory cells mean an LSTM layer has four times as many parameters as an RNN with the same hidden vector size. This paper addresses the vanishing gradient problem using a high order RNN (HORNN) which has additional connections from multiple previous time steps. Speech recognition experiments using British English multi-genre broadcast (MGB3) data showed that the proposed HORNN architectures for rectified linear unit and sigmoid activation functions reduced word error rates (WER) by 4.2% and 6.3% over the corresponding RNNs, and gave similar WERs to a (projected) LSTM while using only 20%--50% of the recurrent layer parameters and computation.
BibTeX
@inproceedings{zhang2018high,
  author = {Chao Zhang and Philip C. Woodland},
  title = {{High Order Recurrent Neural Networks for Acoustic Modelling}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  doi = {10.1109/icassp.2018.8461608},
}
Figure