Basins of Attraction for Electronic Neural Networks

Charles M. Marcus, Robert M. WesterveltNeurIPS 1987

Introduction

This 1988 paper explores the physical dynamics of electronic analog neural networks, revealing that while overloading memories leads to irregular "basins of attraction," the real danger is hardware time delay. The authors demonstrate that time delays combined with "frustrated" network loops cause unwanted oscillations, but these can be neutralized by strategically "diluting" (thinning out) the network connections.

The Challenge of Physical Hardware

In the late 80s, neural network theory often relied on idealized assumptions: neurons switched instantly, and signals propagated across the network without delay. However, real-world hardware—whether biological or electronic—is bound by the laws of physics. The authors tackle two main challenges:

  1. Basin Geometry: In an associative memory, a "well-behaved" basin of attraction is a smooth region of initial states that all lead to the correct memory. The authors wanted to visualize these regions in a physical system pushed to its limits.
  2. Temporal Instability: Real circuits have propagation delays. In symmetric networks (which are mathematically guaranteed to settle into fixed points when there is no delay), these delays can suddenly trigger collective, sustained oscillations. The paper seeks to understand why this happens and how to prevent it.

Experimental Setup

The researchers built a custom electronic analog neural network consisting of eight "neurons" (non-linear sigmoid amplifiers with variable gain) and an interconnection matrix allowing for three-level weights (+1, 0, -1). To study temporal instability, they integrated Charge Coupled Devices (CCDs) into each neuron to introduce adjustable time delays (0.4 to 8 ms).

Block diagram of the experimental setup

A novel basin mapping circuit periodically disconnected the feedback loop, loaded "raster-scanned" initial voltages onto the neurons, and then reconnected the loop to see which memory the network settled into.

Mapping the Basins

The network was programmed with up to four orthogonal memory vectors using a clipped Hebb rule. The results were plotted in 2D slices on a storage oscilloscope to visualize the shape of the basins.

Basins of attraction for three neurons

While the 8-neuron network could technically hold four memories without creating spurious memories, the shape of the basins became highly irregular once three or more memories were stored. Instead of neat quadrants, the boundaries became jagged, meaning small changes in initial state could lead to wildly different results.

Frustration, Cliques, and the Cure of Dilution

The authors found that oscillations only occur in the presence of "frustration"—a concept borrowed from spin glass physics where connections in a loop conflict (e.g., a loop where the product of the weights is negative). While simple loops didn't always oscillate, "frustrated cliques" (densely connected groups where every sub-triangle is frustrated) were highly unstable. The larger the clique, the smaller the time delay needed to trigger a total system oscillation. For a frustrated clique of size $r=8$, the critical delay required to cause oscillation was less than 1/30th of the network's RC time constant.

The most surprising insight is that "diluting" the network—randomly removing connections—breaks up these frustrated cliques much faster than it degrades the actual memory capacity. In a fully connected 8-neuron network with a 0.4 ms delay, the system was plagued by oscillations. By randomly diluting the connections to a connectance of $p \approx 0.85$, the authors successfully "closed" the oscillatory basins and recovered a stable associative memory. Furthermore, diluting a network programmed with the Hebb rule is more efficient at breaking up frustration than diluting a purely random network.

Limitations and Future Work

The study was limited to a very small network (8 neurons), which allowed for clear visualization but may not capture the full complexity of massive VLSI (Very Large Scale Integration) systems. While the authors found a way to "kill" oscillations before memory capacity was compromised, they did not deeply explore the upper limits of this trade-off in larger networks. Future work suggested by the paper involves understanding how these "dangerous" configurations scale as network size increases and applying these stabilization techniques to more complex, non-symmetric architectures.

Summary updated

BibTeX
@inproceedings{marcus1987basins,
  author = {Charles M. Marcus and Robert M. Westervelt},
  title = {{Basins of Attraction for Electronic Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1987},
}
Figure