The Recurrent Temporal Restricted Boltzmann Machine

Ilya Sutskever, Geoffrey E. Hinton, Graham W. TaylorNeurIPS 2008 · OpenReview

Overview

The Recurrent Temporal Restricted Boltzmann Machine (RTRBM) simplifies the complex Temporal RBM by replacing stochastic hidden states with deterministic "mean-field" values. This clever modification transforms a computationally nightmarish inference problem into a straightforward forward pass, allowing the model to be trained effectively using backpropagation through time (BPTT) and Contrastive Divergence.

The Challenge of High-Dimensional Sequences

Modeling high-dimensional sequences—like the fluid motion of a human body or the pixels in a video—is a core challenge in machine learning. The Temporal Restricted Boltzmann Machine (TRBM) was initially a promising candidate for this task, but it had a fatal flaw: inference was nearly impossible.

In a standard TRBM, the hidden state at one timestamp depends stochastically on the hidden state of the previous timestamp. To figure out the "best" hidden state for a given sequence, you would theoretically need to calculate the ratio of two RBM partition functions—a task that is exponentially expensive. Because of this, researchers had to rely on shaky heuristics that didn't always lead to optimal learning.

The RTRBM Architecture

To solve this, the authors introduce the Recurrent TRBM (RTRBM). The key innovation is a subtle but powerful shift in how hidden units are handled across time.

Graphical representation of the Recurrent Temporal Restricted Boltzmann Machine (RTRBM) architecture.
  1. From Stochastic to Deterministic: In the TRBM, hidden units are sampled as binary values (0 or 1). In the RTRBM, the hidden state used for temporal connections is the mean-field value (the continuous probability that a unit is 1).
  2. RNN Mapping: By making the hidden state a deterministic function of the previous state and the current visible input, the RTRBM essentially becomes a Recurrent Neural Network (RNN).
  3. Hybrid Learning: To train the model, the authors use Backpropagation Through Time (BPTT) to handle the temporal dependencies, combined with Contrastive Divergence (CD)—the standard algorithm for training RBMs—to handle the generative part of the model at each timestep.

This architecture allows the model to "remember" the past through its deterministic hidden states while remaining a powerful generative model capable of dreaming up new data.

Key Insights and Emergent Properties

The Power of Determinism: The critical realization of the RTRBM is that hidden-to-hidden connections do not need to be stochastic to yield a powerful generative model. Making them deterministic solves the inference bottleneck without sacrificing expressiveness.

Emergent Hierarchy: When analyzing the learned weights, the authors found a fascinating structural adaptation. Some hidden units developed very weak connections to the visible pixels but very strong connections to other hidden units.

Visualization of the hidden-to-hidden weight matrix, illustrating the emergent hierarchy where some units specialize in temporal dynamics.

These specific units effectively acted as a second hidden layer, specializing in timing and high-level dynamics rather than immediate visual features.

Information Bottleneck: The RTRBM outperformed the TRBM primarily because it learned to "shuttle" more information through its hidden-to-hidden connections. The TRBM's stochastic nature made it harder to maintain a stable "memory" of the past, whereas the RTRBM's deterministic states provided a reliable information conduit.

Experimental Results

The researchers evaluated the RTRBM on two high-dimensional sequence tasks:

  • Bouncing Balls: A synthetic dataset of three balls bouncing in a box (30x30 pixel resolution). The RTRBM successfully learned basic physics; samples showed balls moving with realistic momentum in straight lines, whereas the older TRBM produced balls that moved in jittery "random walks."
  • Motion Capture (MoCap): Real-world human motion data representing joint angles and spatial rotations (49 dimensions). The RTRBM produced much smoother sequences, successfully avoiding common artifacts like "foot-skate" (where a character's feet appear to slide on the floor) and "sticking" that plagued the original TRBM.

Ultimately, because inference in the RTRBM is just a forward pass, the model proved significantly easier and more stable to train than its predecessor.

Limitations and Future Work

Despite its successes, the RTRBM shares some limitations common to early recurrent models:

  • The Vanishing Gradient: Like all standard RNNs, the RTRBM is susceptible to the vanishing gradient problem, making it difficult to learn dependencies over very long time horizons. Integrating techniques like Long Short-Term Memory (LSTM) could potentially address this.
  • Qualitative Evaluation: Because calculating the exact log-likelihood of an RBM is intractable, the evaluation remains largely qualitative (e.g., visual inspection of generated videos). Developing more robust quantitative metrics for generative sequence models is an ongoing challenge.
  • Complexity: While simpler than the TRBM, the model still requires careful tuning of the Contrastive Divergence steps and learning rates across the hybrid BPTT/CD process.

Summary updated

Abstract

The Temporal Restricted Boltzmann Machine (TRBM) is a probabilistic model for sequences that is able to successfully model (i.e., generate nice-looking samples of) several very high dimensional sequences, such as motion capture data and the pixels of low resolution videos of balls bouncing in a box. The major disadvantage of the TRBM is that exact inference is extremely hard, since even computing a Gibbs update for a single variable of the posterior is exponentially expensive. This difficulty has necessitated the use of a heuristic inference procedure, that nonetheless was accurate enough for successful learning. In this paper we introduce the Recurrent TRBM, which is a very slight modification of the TRBM for which exact inference is very easy and exact gradient learning is almost tractable. We demonstrate that the RTRBM is better than an analogous TRBM at generating motion capture and videos of bouncing balls.
BibTeX
@inproceedings{sutskever2008recurrent,
  author = {Ilya Sutskever and Geoffrey E. Hinton and Graham W. Taylor},
  title = {{The Recurrent Temporal Restricted Boltzmann Machine}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2008},
  url = {https://openreview.net/forum?id=ryVAzwb_-B},
}
Figure