How to Construct Deep Recurrent Neural Networks

Razvan Pascanu, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio

Abstract

While recurrent neural networks (RNNs) are inherently "deep" in time, their internal step-by-step transitions are often surprisingly shallow. This paper proposes making RNNs truly deep by replacing simple linear projections with multi-layer perceptrons (MLPs) in the state transitions and output functions. This structural deepening leads to significant performance gains in complex sequence tasks like music and language modeling.

Introduction

In the realm of feedforward neural networks, "depth" is straightforward: it is the number of nonlinear layers between the input and the output. In RNNs, however, the definition is murkier. Because an RNN is unfolded over time, a signal from ten steps ago has passed through ten nonlinearities—making it "deep" temporally.

A standard RNN unfolded in time

However, the authors argue that the transition from one hidden state to the next ($h_{t-1} \to h_t$) is usually "shallow," typically consisting of a single affine transformation followed by a nonlinearity. This limits the model's ability to learn complex, rapidly changing dynamics. The challenge addressed in this paper is identifying exactly where an RNN can be made deeper structurally, and how to do so without making the model impossible to train.

Proposed Architectures

The authors identify three specific "knobs" or functions within an RNN that can be deepened:

  1. Input-to-Hidden: Processing the input $x_t$ through multiple layers before it interacts with the recurrent state.
  2. Hidden-to-Hidden Transition: Using an MLP to compute the next state $h_t$ from the previous state $h_{t-1}$ and input $x_t$. This architecture is termed a DT-RNN (Deep Transition RNN).
  3. Hidden-to-Output: Using an MLP to compute the output $y_t$ from the hidden state.
Comparison of RNN architectures

To address the difficulty of training these deeper structures—specifically the vanishing gradient problem—they introduce shortcut connections (denoted as DT(S)-RNN). These connections provide a "high-speed" linear path for gradients to flow through the MLP transitions, easing optimization.

Furthermore, they introduce a conceptual framework called Neural Operators, viewing the RNN as a combination of a "plus" operator ($\oplus$) for state updates and a "predict" operator ($\rhd$) for outputs. By implementing these operators as MLPs, the deep architecture emerges naturally.

Experimental Setup

The researchers evaluated their models on two main types of sequence modeling tasks:

  • Polyphonic Music Prediction: Using the Nottingham, JSB Chorales, and MuseData datasets. These tasks involve predicting the next set of notes in a complex musical sequence.
  • Language Modeling: Using the Penn Treebank (PTB) corpus for both character-level and word-level prediction.

Key Insights and Results

The deep variants consistently outperformed conventional, shallow RNNs across all evaluated tasks.

  • Temporal Depth vs. Computational Depth: An RNN can be deep in time but shallow in its logic. By adding layers to the transition function, the model can better "disentangle" factors of variation in the input data at each individual time step. A deep transition allows the hidden state to undergo more complex transformations before summarizing the past.
  • Orthogonality to Stacking: This approach is fundamentally different from "Stacked RNNs" (where multiple RNN layers are piled on top of each other). In fact, the two concepts are orthogonal and can be combined: one could design a stacked RNN where each individual layer also utilizes deep internal transitions.
  • Language Modeling: On the Penn Treebank word-level task, the DOT(S)-RNN (Deep Output, Deep Transition with shortcuts) achieved a perplexity of 107.5, significantly better than the shallow RNN (117.7) and even outperforming larger LSTM models of that era.
  • Music Prediction: In almost every musical dataset, at least one of the deep variants (DT-RNN or DOT-RNN) outperformed the standard RNN.
  • Modern Regularization: The authors found that combining these deep architectures with techniques like Dropout, Maxout units, and ReLU activations significantly boosted performance, suggesting that deep RNNs benefit from the same regularization strategies as deep feedforward nets.

Limitations and Future Work

While powerful, the authors acknowledge that training deep RNNs is significantly harder than training shallow ones. Even with shortcut connections, the optimization landscape becomes more complex. Some models required careful initialization or "layer-wise pre-training" (initializing a deep model with the weights of a trained shallow one) to achieve the best results.

Ultimately, the paper opens up a massive design space. Future work involves finding the optimal balance between stacking layers (vertical depth) and deepening transitions (recurrent depth) for specific types of data and tasks.

Summary updated

Abstract

In this paper, we propose a novel way to extend a recurrent neural network (RNN) to a deep RNN. We start by arguing that the concept of the depth in an RNN is not as clear as it is in feedforward neural networks. By carefully analyzing and understanding the architecture of an RNN, we define three points which may be made deeper; (1) input-to-hidden function, (2) hidden-to-hidden transition and (3) hidden-to-output function. This can be considered in addition to stacking multiple recurrent layers proposed earlier by Schmidhuber (1992). Based on this observation, we propose two novel architectures of a deep RNN and provide an alternative interpretation of these deep RNN's using a novel framework based on neural operators. The proposed deep RNN's are empirically evaluated on the tasks of polyphonic music prediction and language modeling. The experimental result supports our claim that the proposed deep RNN's benefit from the depth and outperform the conventional, shallow RNN.
BibTeX
@inproceedings{pascanu2014how,
  author = {Razvan Pascanu and Caglar Gulcehre and KyungHyun Cho and Yoshua Bengio},
  title = {{How to Construct Deep Recurrent Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2014},
  url = {https://openreview.net/forum?id=L80PLIixPIXTH},
}
Figure