Associative Memory in a Network of 'Biological' Neurons

Wulfram GerstnerNeurIPS 1990 · OpenReview

This paper, published in 1990 by Wulfram Gerstner, represents a pivotal moment in computational neuroscience: the attempt to reconcile the elegant, abstract mathematics of the Hopfield network with the messy, "spiky" reality of biological neurons.

The Gap Between Artificial and Biological Memory

In the early 1980s, John Hopfield introduced a revolutionary model of neural networks that functioned as associative memory—the ability of a network to "fill in the blanks" of a partial memory. However, Hopfield’s neurons were "formal" units: they were either ON or OFF (binary) or had a simple graded response.

Real biological neurons do not work that way. They communicate via discrete electrical pulses called spikes, they are subject to significant noise, and they have a refractory period—a "cool-down" time after firing during which they cannot fire again. Gerstner asks a fundamental question: If we replace the artificial units in a Hopfield network with these "real" biological neurons, does the associative memory still work, or does the system collapse into noise?

Modeling the Biological Neuron

Gerstner constructs a model that sits between the extreme complexity of the Hodgkin-Huxley equations and the extreme simplicity of binary units.

  1. Probabilistic Spiking: Instead of a simple threshold, he uses a probabilistic spiking rate. The probability of a neuron firing increases exponentially as the membrane voltage exceeds a threshold.
  2. Refractory Dynamics: He introduces a negative "refractory field" that is added to the potential after every spike. This simulates both the absolute refractory period (where firing is impossible) and the relative refractory period (where firing is just harder).
Refractory function of a standard neuron
  1. The Network: He uses a fully connected network where synaptic strengths are determined by the standard Hebbian learning rule (neurons that fire together, wire together) to store a set of random binary patterns.
  2. Analytical Approach: He solves the model mathematically for the limit of a very large network, allowing him to predict how the "overlap" (the similarity between the current network state and a stored memory) evolves over time.

Key Insights: Robustness and the "Undecided" State

The most important takeaway is that associative memory is a robust property. You don't need perfectly clean, binary neurons to create a system that can retrieve memories; the collective behavior of the network overcomes the "imperfections" of individual biological neurons.

However, introducing biological constraints reveals new, complex behaviors:

  • The Hopfield Limit: Gerstner showed that if you shrink the refractory period to zero, his model mathematically reduces back to the original Hopfield solution.
  • Critical Refractory Period: He identified a critical duration for the refractory period. Below this value, the network behaves like a standard Hopfield model. Above it, the qualitative behavior changes.
  • The "Undecided" State: In a standard Hopfield model, the network is almost always "forced" to choose a memory, even if the input is total gibberish. Gerstner found that with a sufficiently long refractory period, a trivial state (zero overlap) becomes stable. This means the network can remain "undecided" if the input doesn't strongly resemble a known memory.
Overlap as a function of temperature for different refractory periods

As shown in the phase diagrams, the biological model exhibits a sharper drop-off in retrieval ability as noise (temperature) increases compared to the artificial model. For certain noise levels and longer refractory durations, the network exhibits bistability. It can either stay at zero overlap or jump to a high-overlap retrieval state, depending on how strong the initial "hint" or cue is.

Limitations and Future Directions

The author acknowledges several "unbiological" simplifications that remained in this model:

  • Full Connectivity: In the brain, neurons are not connected to every other neuron.
  • Symmetry: The model assumes the connection from neuron A to B is the same as B to A, which is rarely true in biology.
  • Linear Learning: The Hebbian rule used is a mathematical idealization.

Gerstner suggests that the next steps involve moving toward sparse connectivity and non-linear learning rules. Ultimately, while biological details add complexity to artificial models, they also add "interesting features to the variety of states available to the network"—a nod toward the complex dynamics required for actual cognition.

Summary updated

Abstract

The Hopfield network (Hopfield, 1982,1984) provides a simple model of an associative memory in a neuronal structure. This model, however, is based on highly artificial assumptions, especially the use of formal-two state neu(cid:173) rons (Hopfield, 1982) or graded-response neurons (Hopfield, 1984). \Vhat happens if we replace the formal neurons by 'real' biological neurons? \Ve address this question in two steps. First, we show that a simple model of a neuron can capture all relevant features of neuron spiking, i. e., a wide range of spiking frequencies and a realistic distribution of interspike inter(cid:173) vals. Second, we construct an associative memory by linking these neurons together. The analytical solution for a large and fully connected network shows that the Hopfield solution is valid only for neurons with a short re(cid:173) fractory period. If the refractory period is longer than a crit.ical duration ie, the solutions are qualitatively different. The associative character of the solutions, however, is preserved.
BibTeX
@inproceedings{gerstner1990associative,
  author = {Wulfram Gerstner},
  title = {{Associative Memory in a Network of 'Biological' Neurons}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
  url = {https://openreview.net/forum?id=HJboo_-_ZB},
}
Figure