An EM Based Training Algorithm for Recurrent Neural Networks

Jan Unkelbach, Yi Sun, Jürgen SchmidhuberICANN 2009 · DOI

TL;DR

This paper introduces a novel Expectation-Maximization (EM) based training algorithm for recurrent neural networks (RNNs) that can model stochastic dynamical systems. By treating RNNs as nonlinear state-space models and employing a particle smoother for approximate inference, the method allows for the modeling of process noise, a crucial aspect often overlooked in conventional RNN training.

What Problem Does This Paper Address?

Recurrent neural networks (RNNs) are powerful tools for modeling sequential data and identifying dynamical systems. However, traditional training methods for RNNs, which typically involve minimizing a quadratic error, implicitly assume that the underlying system is deterministic. This means they don't account for "process noise" – inherent randomness in the system's internal state transitions. This limitation makes them less suitable for modeling real-world systems where such noise is prevalent, like biological processes or financial markets. State-space models, on the other hand, can explicitly model this noise, but learning complex nonlinear state-space models can be computationally challenging. This paper bridges this gap by proposing a method to train RNNs that can effectively model stochastic dynamical systems, akin to more complex state-space models, but with the flexibility of RNNs.

Methods

The core of the paper is an EM-based training algorithm adapted for RNNs. The authors frame RNNs as nonlinear state-space models. The EM algorithm iteratively refines model parameters by:

  • E-step (Expectation): Estimating the probability distribution of the hidden states given the observed data and current model parameters. Because the nonlinear transfer functions in RNNs make exact inference intractable, they employ a Particle Smoother. This is a sequential Monte Carlo method that uses a particle filter to approximate the forward pass (estimating state probabilities at each time step) and a backward recursion to refine these estimates, providing a smoothed distribution over the hidden states.
  • M-step (Maximization): Updating the model parameters (weights and noise covariances) to maximize the expected log-likelihood of the data, using the smoothed state distributions from the E-step. The authors leverage the structural similarity between their RNN formulation and linear dynamical systems (LDS) to derive update rules for the parameters, particularly for the weight matrices.

Datasets

The paper uses two types of data for demonstration:

  • Synthetic Data: Generated by a recurrent network with known parameters, allowing for a controlled evaluation of the proposed method. This synthetic data includes both process and measurement noise.
  • Breathing Motion Data: A real-world dataset representing the motion of a lung tumor during respiration. This data is characterized by significant variations in amplitude and period, and importantly, the authors note that its variability is primarily due to process noise (breathing variations) rather than measurement noise.

Key Insights and Ideas

  • RNNs as Generative Models for Stochastic Systems: The paper successfully demonstrates that RNNs, when trained with this EM-based approach, can act as generative models for systems exhibiting process noise, going beyond the deterministic assumptions of standard training.
  • Particle Smoother as a Key Enabler: The particle smoother is crucial for making the EM algorithm tractable for nonlinear RNNs. It provides a practical way to approximate the complex posterior distributions required for the E-step.
  • Leveraging LDS Structure: The authors cleverly exploit the structural similarities between their nonlinear state-space RNN formulation and linear dynamical systems to derive efficient update rules for the M-step, simplifying parameter learning.
  • Modeling Real-World Variability: The application to breathing motion data highlights the practical relevance of modeling process noise, as this real-world phenomenon is inherently stochastic.

Key Results

  • Successful Training on Synthetic Data: The trained RNN on synthetic data was able to learn the underlying harmonic oscillation and accurately estimate both process and measurement noise levels. The prediction performance was comparable to using the true generative process.
  • Breathing Motion Prediction: For the breathing motion data, the trained network captured the oscillatory behavior. While it showed promise in predicting the motion, the authors note that the estimated measurement noise was too large, suggesting the network didn't perfectly capture the dynamics and wasn't an adequate generative model in this specific case.
  • Efficiency of Particle Smoother: The results were obtained with a relatively small number of particles (N=100, and even N=10), indicating the particle smoother's efficiency for this task.

Limitations and Future Work

  • Local Minima: The authors acknowledge that dynamical system identification is prone to local minima, a challenge that also affects linear models and standard RNN training.
  • Breathing Motion Model Adequacy: For the breathing motion data, the trained network's ability to precisely model the dynamics was limited, leading to an overestimation of measurement noise and a less adequate generative model. This suggests further refinement might be needed for highly complex or irregular real-world signals.
  • Future Work: The paper suggests further investigation into the performance characterization of the training algorithm, the influence of the number of particles on accuracy, and comparisons with standard RNN training methods. They also mention that the particle numbers used, while effective, might not fully characterize the posterior distribution at every time step, indicating a need for more in-depth analysis of particle count effects.

Summary updated

BibTeX
@inproceedings{unkelbach2009em,
  author = {Jan Unkelbach and Yi Sun and Jürgen Schmidhuber},
  title = {{An EM Based Training Algorithm for Recurrent Neural Networks}},
  booktitle = {ICANN},
  year = {2009},
  doi = {10.1007/978-3-642-04274-4\_99},
}
Figure