Time-Sequential Self-Organization of Hierarchical Neural Networks

Ronald H. Silverman, Andrew S. NoetzelNeurIPS 1987

This 1988 paper offers a fascinating look at how researchers approached the "credit assignment problem" in the early days of neural network research, before backpropagation became the undisputed king of the field. The authors propose a method for training multi-layered neural networks without backpropagation or explicit labels. By training layers one at a time using a "lesson plan" of increasingly complex patterns, the network can autonomously learn to extract high-level concepts like orientation.

The Credit Assignment Problem

In the late 1980s, a major hurdle in neural network research was determining how to adjust weights in "hidden" layers—those not directly connected to the input or the final output.

While backpropagation (using error signals from the output to update internal layers) was emerging as a solution, the authors argued it required specialized neural pathways for feedback that might not be biologically plausible or computationally simple. They sought a purely feed-forward, unsupervised way for a hierarchy of neurons to organize itself, where each layer learns to represent features of the layer below it without needing a global error signal.

Hierarchical Architecture and Time-Sequential Training

To avoid backpropagation, the authors introduce a "time-sequential" approach to self-organization. Instead of training the whole network at once, they train one layer, "freeze" its weights, and then use its outputs to train the next layer.

Hierarchical multi-layer network architecture

This layer-by-layer approach relies heavily on an external "instructor" or pattern generator that provides a Lesson Plan. This is the most novel conceptual part of the framework. The instructor provides sets of patterns that increase in complexity over time. Once a lower layer learns simple features (like dots or lines), the instructor provides combinations of those features to help the next layer learn abstractions (like orientation).

Neuron Mechanics: Hebbian Learning and Lateral Inhibition

To make this unsupervised learning work, the individual neurons rely on a few specific biological inspirations:

  • Capacitive Synapses: Unlike standard models where neurons react to instantaneous input, these neurons integrate excitation over time. This allows the network to gather a statistical measure of stochastic input, making the network more robust to noise. A single noisy frame won't trigger a weight change; only consistent, repeated patterns will.
  • Modified Hebbian Learning: When a neuron fires, weights are adjusted based on whether their specific contribution was above or below the mean activity. The authors use a unique sine-function to determine the magnitude of the change. This creates a "sweet spot" for learning in the middle of the weight range and ensures that weights near 0 or 1 (saturation) change slowly, leading to more stable end-states and preventing catastrophic forgetting.
  • Lateral Inhibition: When one neuron in a layer fires, it temporarily zeros out the excitation of its neighbors.
Neuron model demonstrating summation and lateral inhibition

This lateral inhibition acts as a form of "competitive learning." It ensures that different neurons capture different patterns rather than every neuron in a layer learning the exact same feature.

Experiments and Key Results

The paper uses synthetic, simplified datasets for simulation, specifically 4x4 and 6x6 binary grids where "active" pixels represent lines. The lesson plan consists of individual rows or columns (simple patterns) followed by subsets of multiple vertical or horizontal lines (complex patterns).

The authors successfully simulated a two-layer hierarchy that learned to distinguish between horizontal and vertical orientations:

  1. Layer 1 learned to respond to specific, individual lines at specific positions.
  2. Layer 2 (the uppermost layer) developed position-independent sensitivity. After training, a specific neuron would fire if any vertical line appeared anywhere on the grid, while another would fire for any horizontal line.
  3. Thanks to lateral inhibition, adjacent neurons successfully developed "opposite orientation specificities," mimicking the organization seen in the biological visual cortex.

Limitations and Future Work

While the time-sequential approach successfully demonstrates hierarchical feature extraction without backpropagation, it comes with notable limitations:

  • The Burden of the Instructor: The authors admit the central difficulty is designing the lesson plan. The human designer must know what the intermediate features should be, which shifts the complexity from labeling data to curating a curriculum.
  • Control Signals: While the method avoids backpropagation, it still requires a global "control signal" to tell a layer when it has finished training and should freeze its weights.
  • Scaling: The simulations were performed on very small grids. Whether this time-sequential approach scales to complex, real-world images remains an open question not addressed in the text.

Summary updated

BibTeX
@inproceedings{silverman1987time,
  author = {Ronald H. Silverman and Andrew S. Noetzel},
  title = {{Time-Sequential Self-Organization of Hierarchical Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1987},
}
Figure