Induction of Multiscale Temporal Structure

Michael Mozer

TL;DR

Standard recurrent neural networks (RNNs) are notoriously "short-sighted," struggling to link events separated by long time gaps. This paper introduces a simple but effective fix: equipping hidden units with individual time constants that allow them to retain information over different scales. This effectively creates a "bird's eye view" of a sequence, making global patterns much easier to learn.

The Problem: Short-Sighted Networks

In the early 90s, the connectionist community faced a major hurdle: while RNNs were highly capable of learning local patterns (like predicting the next note in a musical scale), they failed at learning global ones (like recognizing the repetition of a musical theme several bars later).

The core issue is that backpropagation through time (BPTT) struggles to carry error signals across long temporal "gaps." As the interval between a cause and its effect grows, the gradient often vanishes or gets lost in the noise of intervening events. The author identifies this as a failure to induce "multiscale temporal structure"—the ability to see both the trees (local transitions) and the forest (global trends) simultaneously.

Methods: The Reduced Description

To address this, the paper proposes a modification to the standard RNN activation rule. Instead of a unit's state being determined entirely by the current input and the previous state, a time constant ($\tau_i$) is introduced for each context unit $i$:

$$c_i(t) = \tau_i c_i(t-1) + (1 - \tau_i) \tanh[net_i(t)]$$

Here is the intuition behind the math:

  • When $\tau = 0$: The unit behaves like a standard RNN unit. It is highly responsive and changes its state immediately based on new input.
  • When $\tau$ is large (close to 1): The unit becomes "sluggish." It holds onto its previous value and only slowly incorporates new information.

By utilizing a mix of units with different $\tau$ values, the network creates a Reduced Description of the input.

Reduced description of a sequence

The high-$\tau$ units act as low-pass filters, smoothing out the "noise" of individual time steps to represent the overall contour of the sequence. Crucially, the paper shows mathematically that a large $\tau$ allows the error signal to flow much further back in time without decaying, facilitating the learning of long-range dependencies.

Key Insights

  • Smoothing as Compression: By averaging inputs over time, high-$\tau$ units effectively "compress" the sequence. This is analogous to listening to an audio recording at double speed; you lose the individual notes, but the overarching melodic arc becomes unmistakable.
  • The Gradient "Bridge": The time constant acts as a physical bridge for the gradient. In standard BPTT, the error signal has to pass through many non-linearities and weights, which degrades it. In this model, the $\tau$ term provides a linear path for the error to travel back through time.
  • Local vs. Global Specialization: The results suggest that a "hybrid" hidden layer—containing some fast units for local transitions and some slow units for global context—is vastly superior to a uniform architecture.

Experimental Results

The paper evaluates the approach using two synthetic but illustrative benchmarks:

  1. The Trigger-Gap Task: A sequence starts with a "trigger" symbol, followed by a "gap" of $N$ symbols, and ends with the trigger again. The network must predict the second trigger. This is a pure test of long-term memory.
  2. AABA Phrase Patterns: Synthetic musical melodies consisting of four phrases. To predict the start of the final 'A' phrase or the 'END' marker, the network must remember where it is in the global AABA structure.

Closing the Gap

In the Trigger-Gap task, standard RNNs began failing at a gap of only 4 symbols and failed 100% of the time by a gap of 8. The Reduced Description (RD) net, however, achieved a 0% failure rate on gaps up to 10 and scaled linearly to much larger gaps.

Performance comparison on the Trigger-Gap task

Musical Coherence

On the AABA task, the RD net significantly outperformed the standard version on "global" predictions (75.6% vs 58.4% accuracy). Specifically, it was much better at predicting the end of the sequence and the return of the 'A' phrase. When applied to actual music composition (Bach and folk tunes), listeners noted that networks with multiscale units produced pieces with much better "global coherence" compared to the aimless wandering of standard RNNs.

Limitations and Future Work

The primary drawback of this approach is the "Goldilocks" problem: the choice of $\tau$ is critical. If $\tau$ is too small, the network cannot see across the gap; if it is too large, the unit is too sluggish to learn anything useful. Training $\tau$ directly via backpropagation proved difficult because the error surface is riddled with local optima.

The paper concludes by suggesting a dynamic, hybrid approach (developed alongside Jürgen Schmidhuber). This involves a "chunking" architecture where time constants are not fixed but are adjusted dynamically based on the prediction error of a lower-level network. When the lower network is surprised (indicating a boundary between "chunks"), the upper network updates its global representation.

Summary updated

Abstract

Learning structure in temporally-extended sequences is a difficult com(cid:173) putational problem because only a fraction of the relevant information is available at any instant. Although variants of back propagation can in principle be used to find structure in sequences, in practice they are not sufficiently powerful to discover arbitrary contingencies, especially those spanning long temporal intervals or involving high order statistics. For example, in designing a connectionist network for music composition, we have encountered the problem that the net is able to learn musical struc(cid:173) ture that occurs locally in time-e.g., relations among notes within a mu(cid:173) sical phrase-but not structure that occurs over longer time periods--e.g., relations among phrases. To address this problem, we require a means of constructing a reduced deacription of the sequence that makes global aspects more explicit or more readily detectable. I propose to achieve this using hidden units that operate with different time constants. Simulation experiments indicate that slower time-scale hidden units are able to pick up global structure, structure that simply can not be learned by standard back propagation. Many patterns in the world are intrinsically temporal, e.g., speech, music, the un(cid:173) folding of events. Recurrent neural net architectures have been devised to accom(cid:173) modate time-varying sequences. For example, the architecture shown in Figure 1 can map a sequence of inputs to a sequence of outputs. Learning structure in temporally-extended sequences is a difficult computational problem because the in(cid:173) put pattern may not contain all the task-relevant information at any instant. Thus, 275
BibTeX
@inproceedings{mozer1991induction,
  author = {Michael Mozer},
  title = {{Induction of Multiscale Temporal Structure}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1991},
  url = {https://openreview.net/forum?id=BJbxtwW_-H},
}
Figure