Pre-training of Recurrent Neural Networks via Linear Autoencoders

Luca Pasa, Alessandro Sperduti

TL;DR

This paper introduces a novel unsupervised pre-training method for Recurrent Neural Networks (RNNs) by initializing their weights using the optimal solutions of linear autoencoders for sequences. By starting the learning process from a mathematically grounded "good neighborhood," the authors effectively bypass the vanishing gradient problem and achieve new state-of-the-art results on complex polyphonic music prediction tasks.

The Challenge with RNNs

Recurrent Neural Networks are theoretically ideal for modeling sequences, but they are notoriously difficult to train. The primary culprit is the vanishing gradient problem: as errors are backpropagated through time, the gradient signal often shrinks to near zero, making it nearly impossible for the network to learn long-term dependencies.

While researchers have developed architectural workarounds (like LSTMs) or advanced optimization techniques (like Hessian-Free Optimization), the authors argue that the starting point of the weights matters immensely. Most RNNs are initialized with random weights, which often places the model in a region of the parameter space where it struggles to capture even basic temporal patterns.

Proposed Method: Linear Initialization

The authors propose a two-step approach: first, solve a linear version of the problem, then use that solution to jump-start the non-linear RNN.

  1. Linear Autoencoders for Sequences: The authors model the sequence using a Linear Dynamical System (LDS). They treat the hidden state as an encoding of the input history. They derive a closed-form solution to find the optimal weight matrices ($A$ and $B$) that allow the system to reconstruct the input sequence as accurately as possible.
  2. The PCA Connection: They show that finding these optimal weights is essentially equivalent to performing Principal Component Analysis (PCA) on the "unrolled" history of the sequence. This ensures the hidden units capture the directions of highest variance in the data.
  3. SVFORBIGDATA Algorithm: Because performing SVD (Singular Value Decomposition) on massive sequence datasets is computationally prohibitive, the authors developed an incremental, approximate algorithm. It processes the data in "slices," updating the principal components as it goes, making the method scalable to real-world datasets.
Algorithms for approximating SVD on large datasets
  1. RNN Initialization:
    • The input-to-hidden weights ($W_{input}$) are set to the optimal linear matrix $A$.
  2. The hidden-to-hidden weights ($W_{hidden}$) are set to the optimal linear matrix $B$.
  3. The output weights ($W_{output}$) are initialized using a simple least-squares solution (pseudoinverse) based on the initial hidden states.
  4. Finally, the whole network is fine-tuned using standard stochastic gradient descent.

Key Insights

  • Linearity as a Foundation: Even though RNNs are non-linear, a well-optimized linear model can capture the "skeleton" of temporal dependencies. Initializing with this skeleton allows the non-linearities to focus on refining the model rather than learning basic structure from scratch.
  • Uncorrelated Hidden States: By using a PCA-like approach for initialization, the authors ensure that the hidden units start off representing uncorrelated features of the input history, maximizing the information capacity of the hidden layer.
  • Efficiency: The pre-training step is remarkably fast compared to the thousands of epochs required for standard RNN training. It provides a massive "head start" for a very low computational cost.

Datasets and Results

The approach was evaluated on four well-known benchmarks for polyphonic music modeling: Nottingham (folk tunes), Piano-midi.de (classical piano), MuseData (orchestral/choral), and JSB Chorales (J.S. Bach). These datasets are challenging because they require predicting which notes (out of 88 possible piano keys) will be played next based on complex harmony and rhythm histories.

The results were significant, often doubling the accuracy of previous models:

  • State-of-the-Art Performance: The pre-trained RNN (PreT-RNN) outperformed the then-standard RNN-RBM (Restricted Boltzmann Machine) on three of the four datasets.
  • JSB Chorales: PreT-RNN achieved 65.67% accuracy, nearly doubling the 33.12% achieved by the RNN-RBM.
  • MuseData: Accuracy jumped to 57.57% compared to the previous best of 34.02%.
  • Overcoming Randomness: While randomly initialized RNNs (even with many hidden units) struggled to improve significantly over time, the pre-trained models showed an immediate and sharp improvement in performance from the very first epoch.
Training and test accuracy curves across four polyphonic music datasets

Limitations and Future Work

  • Linear Assumptions: The method relies on the assumption that a linear approximation of the sequence dynamics is useful. For extremely chaotic or purely symbolic sequences where linear correlations are weak, this initialization might be less effective.
  • Symmetric Sigmoids: The authors noted that their method works best with symmetric sigmoidal activation functions (which behave linearly near the origin), suggesting the initialization might need adjustment for other activations like ReLU.
  • Future Directions: The authors suggest extending this framework to more complex structures, such as tree-structured data or deeper recurrent architectures.

Summary updated

Abstract

We propose a pre-training technique for recurrent neural networks based on linear autoencoder networks for sequences, i.e. linear dynamical systems modelling the target sequences. We start by giving a closed form solution for the definition of the optimal weights of a linear autoencoder given a training set of sequences. This solution, however, is computationally very demanding, so we suggest a procedure to get an approximate solution for a given number of hidden units. The weights obtained for the linear autoencoder are then used as initial weights for the input-to-hidden connections of a recurrent neural network, which is then trained on the desired task. Using four well known datasets of sequences of polyphonic music, we show that the proposed pre-training approach is highly effective, since it allows to largely improve the state of the art results on all the considered datasets.
BibTeX
@inproceedings{pasa2014pre,
  author = {Luca Pasa and Alessandro Sperduti},
  title = {{Pre-training of Recurrent Neural Networks via Linear Autoencoders}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
}
Figure