TL;DR
Standard recurrent neural networks (RNNs) are notoriously "short-sighted," struggling to link events separated by long time gaps. This paper introduces a simple but effective fix: equipping hidden units with individual time constants that allow them to retain information over different scales. This effectively creates a "bird's eye view" of a sequence, making global patterns much easier to learn.
The Problem: Short-Sighted Networks
In the early 90s, the connectionist community faced a major hurdle: while RNNs were highly capable of learning local patterns (like predicting the next note in a musical scale), they failed at learning global ones (like recognizing the repetition of a musical theme several bars later).
The core issue is that backpropagation through time (BPTT) struggles to carry error signals across long temporal "gaps." As the interval between a cause and its effect grows, the gradient often vanishes or gets lost in the noise of intervening events. The author identifies this as a failure to induce "multiscale temporal structure"—the ability to see both the trees (local transitions) and the forest (global trends) simultaneously.
Methods: The Reduced Description
To address this, the paper proposes a modification to the standard RNN activation rule. Instead of a unit's state being determined entirely by the current input and the previous state, a time constant ($\tau_i$) is introduced for each context unit $i$:
$$c_i(t) = \tau_i c_i(t-1) + (1 - \tau_i) \tanh[net_i(t)]$$
Here is the intuition behind the math:
- When $\tau = 0$: The unit behaves like a standard RNN unit. It is highly responsive and changes its state immediately based on new input.
- When $\tau$ is large (close to 1): The unit becomes "sluggish." It holds onto its previous value and only slowly incorporates new information.
By utilizing a mix of units with different $\tau$ values, the network creates a Reduced Description of the input.
The high-$\tau$ units act as low-pass filters, smoothing out the "noise" of individual time steps to represent the overall contour of the sequence. Crucially, the paper shows mathematically that a large $\tau$ allows the error signal to flow much further back in time without decaying, facilitating the learning of long-range dependencies.
Key Insights
- Smoothing as Compression: By averaging inputs over time, high-$\tau$ units effectively "compress" the sequence. This is analogous to listening to an audio recording at double speed; you lose the individual notes, but the overarching melodic arc becomes unmistakable.
- The Gradient "Bridge": The time constant acts as a physical bridge for the gradient. In standard BPTT, the error signal has to pass through many non-linearities and weights, which degrades it. In this model, the $\tau$ term provides a linear path for the error to travel back through time.
- Local vs. Global Specialization: The results suggest that a "hybrid" hidden layer—containing some fast units for local transitions and some slow units for global context—is vastly superior to a uniform architecture.
Experimental Results
The paper evaluates the approach using two synthetic but illustrative benchmarks:
- The Trigger-Gap Task: A sequence starts with a "trigger" symbol, followed by a "gap" of $N$ symbols, and ends with the trigger again. The network must predict the second trigger. This is a pure test of long-term memory.
- AABA Phrase Patterns: Synthetic musical melodies consisting of four phrases. To predict the start of the final 'A' phrase or the 'END' marker, the network must remember where it is in the global AABA structure.
Closing the Gap
In the Trigger-Gap task, standard RNNs began failing at a gap of only 4 symbols and failed 100% of the time by a gap of 8. The Reduced Description (RD) net, however, achieved a 0% failure rate on gaps up to 10 and scaled linearly to much larger gaps.
Musical Coherence
On the AABA task, the RD net significantly outperformed the standard version on "global" predictions (75.6% vs 58.4% accuracy). Specifically, it was much better at predicting the end of the sequence and the return of the 'A' phrase. When applied to actual music composition (Bach and folk tunes), listeners noted that networks with multiscale units produced pieces with much better "global coherence" compared to the aimless wandering of standard RNNs.
Limitations and Future Work
The primary drawback of this approach is the "Goldilocks" problem: the choice of $\tau$ is critical. If $\tau$ is too small, the network cannot see across the gap; if it is too large, the unit is too sluggish to learn anything useful. Training $\tau$ directly via backpropagation proved difficult because the error surface is riddled with local optima.
The paper concludes by suggesting a dynamic, hybrid approach (developed alongside Jürgen Schmidhuber). This involves a "chunking" architecture where time constants are not fixed but are adjusted dynamically based on the prediction error of a lower-level network. When the lower network is surprised (indicating a boundary between "chunks"), the upper network updates its global representation.
Summary updated