Abstract
While recurrent neural networks (RNNs) are inherently "deep" in time, their internal step-by-step transitions are often surprisingly shallow. This paper proposes making RNNs truly deep by replacing simple linear projections with multi-layer perceptrons (MLPs) in the state transitions and output functions. This structural deepening leads to significant performance gains in complex sequence tasks like music and language modeling.
Introduction
In the realm of feedforward neural networks, "depth" is straightforward: it is the number of nonlinear layers between the input and the output. In RNNs, however, the definition is murkier. Because an RNN is unfolded over time, a signal from ten steps ago has passed through ten nonlinearities—making it "deep" temporally.
However, the authors argue that the transition from one hidden state to the next ($h_{t-1} \to h_t$) is usually "shallow," typically consisting of a single affine transformation followed by a nonlinearity. This limits the model's ability to learn complex, rapidly changing dynamics. The challenge addressed in this paper is identifying exactly where an RNN can be made deeper structurally, and how to do so without making the model impossible to train.
Proposed Architectures
The authors identify three specific "knobs" or functions within an RNN that can be deepened:
- Input-to-Hidden: Processing the input $x_t$ through multiple layers before it interacts with the recurrent state.
- Hidden-to-Hidden Transition: Using an MLP to compute the next state $h_t$ from the previous state $h_{t-1}$ and input $x_t$. This architecture is termed a DT-RNN (Deep Transition RNN).
- Hidden-to-Output: Using an MLP to compute the output $y_t$ from the hidden state.
To address the difficulty of training these deeper structures—specifically the vanishing gradient problem—they introduce shortcut connections (denoted as DT(S)-RNN). These connections provide a "high-speed" linear path for gradients to flow through the MLP transitions, easing optimization.
Furthermore, they introduce a conceptual framework called Neural Operators, viewing the RNN as a combination of a "plus" operator ($\oplus$) for state updates and a "predict" operator ($\rhd$) for outputs. By implementing these operators as MLPs, the deep architecture emerges naturally.
Experimental Setup
The researchers evaluated their models on two main types of sequence modeling tasks:
- Polyphonic Music Prediction: Using the Nottingham, JSB Chorales, and MuseData datasets. These tasks involve predicting the next set of notes in a complex musical sequence.
- Language Modeling: Using the Penn Treebank (PTB) corpus for both character-level and word-level prediction.
Key Insights and Results
The deep variants consistently outperformed conventional, shallow RNNs across all evaluated tasks.
- Temporal Depth vs. Computational Depth: An RNN can be deep in time but shallow in its logic. By adding layers to the transition function, the model can better "disentangle" factors of variation in the input data at each individual time step. A deep transition allows the hidden state to undergo more complex transformations before summarizing the past.
- Orthogonality to Stacking: This approach is fundamentally different from "Stacked RNNs" (where multiple RNN layers are piled on top of each other). In fact, the two concepts are orthogonal and can be combined: one could design a stacked RNN where each individual layer also utilizes deep internal transitions.
- Language Modeling: On the Penn Treebank word-level task, the DOT(S)-RNN (Deep Output, Deep Transition with shortcuts) achieved a perplexity of 107.5, significantly better than the shallow RNN (117.7) and even outperforming larger LSTM models of that era.
- Music Prediction: In almost every musical dataset, at least one of the deep variants (DT-RNN or DOT-RNN) outperformed the standard RNN.
- Modern Regularization: The authors found that combining these deep architectures with techniques like Dropout, Maxout units, and ReLU activations significantly boosted performance, suggesting that deep RNNs benefit from the same regularization strategies as deep feedforward nets.
Limitations and Future Work
While powerful, the authors acknowledge that training deep RNNs is significantly harder than training shallow ones. Even with shortcut connections, the optimization landscape becomes more complex. Some models required careful initialization or "layer-wise pre-training" (initializing a deep model with the weights of a trained shallow one) to achieve the best results.
Ultimately, the paper opens up a massive design space. Future work involves finding the optimal balance between stacking layers (vertical depth) and deepening transitions (recurrent depth) for specific types of data and tasks.
Summary updated