Associative memory in realistic neuronal networks

Peter E. LathamNeurIPS 2001 · OpenReview

Introduction

In classical neural network theory, associative memory is often modeled using "attractor networks," where specific patterns of activity are stable. However, a major discrepancy exists between these abstract Hopfield models and biological reality: real brains maintain a "background state" of low-level, steady firing (typically around a few Hz).

Previous theoretical work suggested that for a stable background and multiple memories to co-exist, neurons must have a "superlinear" (S-shaped) gain function at low firing rates. The problem is that most realistic neurons exhibit linear or sublinear gain functions above a few Hz. Furthermore, most models relied on "sparse coding," where a vanishingly small fraction of neurons participate in a memory. This paper bridges the gap by demonstrating how neuronal networks can maintain associative memories alongside a stable, low-rate background state without requiring complex, non-biological gain functions.

Methods

The author employs a combination of Mean Field Theory and numerical simulations using analog firing-rate neurons. Key methodological features include:

  • Firing Rate Modeling: Instead of complex conductance-based models, the author uses simplified equilibrium equations where a neuron's firing rate is a function of its synaptic inputs.
  • Balanced Network Regime: The model operates in a regime where excitation and inhibition are balanced. This creates a negative feedback loop where increasing the average firing rate actually decreases the effective gain, providing a natural stabilization mechanism.
  • Connectivity Matrix: The network uses a "Hopfield prescription" to embed memories, but adds a clipping function to ensure synaptic weights remain within realistic bounds (non-negative and non-infinite).
  • Scaling: Unlike sparse coding models, this research investigates the case where the number of neurons involved in a memory scales proportionally with the total number of neurons in the network.

Key Insights and Results

The most significant insight of this work is that inhibitory feedback can stabilize attractors even if the neuronal gain functions never saturate. In the sparse coding limit, this feedback is too weak to help; by involving a "reasonable fraction" of neurons in each memory, the network can leverage inhibitory feedback to maintain stability.

Mean field equations confirm that a stable background state (where the memory overlap $w=0$) and a stable memory state ($w>0$) can exist simultaneously in the same network.

Stability of background and memory states

There is a fundamental tradeoff between the firing rate of the "memory" neurons and the stability of the background. If the memory state is pushed to too high a firing rate, it can destabilize the background state through an "exchange of stability." However, for sufficiently large coupling strengths ($\epsilon$), the network's behavior becomes largely independent of the exact strength of the memories, making the system robust to parameter fluctuations.

The model successfully maintains a background firing rate of approximately 3 Hz, matching in vivo observations, while memory states operate at higher frequencies. Simulations with 2,000 neurons showed that memories could be embedded successfully within a specific range of the gain parameter ($\beta$).

Firing rates and stability regions

Furthermore, analysis indicates that the "stability region" for these memories actually grows as the number of neurons ($N$) in the network increases, because the noise fluctuations that destabilize the background scale as $N^{-1/2}$.

Limitations and Future Work

While this paper makes significant strides in biological realism, it has some limitations:

  • Simplified Inhibition: To keep the mean field theory tractable, the author assumed inhibitory neurons were completely homogeneous. Real cortical inhibition is highly diverse in terms of cell types and connectivity.
  • Rate vs. Spikes: The study uses firing-rate models. While useful for studying equilibria, these models may miss fast temporal dynamics. The author notes that future work should involve simulations with spiking neurons.
  • Parameter Exploration: While the results are encouraging, a much larger exploration of the parameter space is needed to ensure these results are truly robust across different biological conditions.

Summary updated

Abstract

Almost two decades ago, Hopfield [1] showed that networks of highly reduced model neurons can exhibit multiple attracting fixed points, thus providing a substrate for associative memory. It is still not clear, however, whether realistic neuronal networks can support multiple attractors. The main difficulty is that neuronal networks in vivo exhibit a stable background state at low firing rate, typ(cid:173) ically a few Hz. Embedding attractor is easy; doing so without destabilizing the background is not. Previous work [2, 3] focused on the sparse coding limit, in which a vanishingly small number of neurons are involved in any memory. Here we investigate the case in which the number of neurons involved in a memory scales with the number of neurons in the network. In contrast to the sparse coding limit, we find that multiple attractors can co-exist robustly with a stable background state. Mean field theory is used to under(cid:173) stand how the behavior of the network scales with its parameters, and simulations with analog neurons are presented. One of the most important features of the nervous system is its ability to perform associative memory. It is generally believed that associative memory is implemented using attractor networks - experimental studies point in that direction [4- 7], and there are virtually no competing theoretical models. Perhaps surprisingly, however, it is still an open theoretical question whether attractors can exist in realistic neu(cid:173) ronal networks. The "realistic" feature that is probably hardest to capture is the steady firing at low rates - the background state - that is observed throughout the intact nervous system [8- 13]. The reason it is difficult to build an attractor network that is stable at low firing rates, at least in the sparse coding limit, is as follows [2,3]: Attractor networks are constructed by strengthening recurrent connections among sub-populations of neurons. The strengthening must be large enough that neurons within a sub-population can sustain a high firing rate state, but not so large that the sub-population can be spontaneously active. This implies that the neuronal gain functions - the firing rate of the post-synaptic neurons as a function of the average • http) / culture.neurobio.ucla.edu/ "'pel firing rate of the pre-synaptic neurons - must be sigmoidal: small at low firing rate to provide stability, high at intermediate firing rate to provide a threshold (at an unstable equilibrium), and low again at high firing rate to provide saturation and a stable attractor. In other words, a requirement for the co-existence of a stable background state and multiple attractors is that the gain function of the excitatory neurons be super linear at the observed background rates of a few Hz [2,3]. However - and this is where the problem lies - above a few Hz most realistic gain function are nearly linear or sublinear (see, for example, Fig. Bl of [14]). The superlinearity requirement rests on the implicit assumption that the activity of the sub-population involved in a memory does not affect the other neurons in the network. While this assumption is valid in the sparse coding limit , it breaks down in realistic networks containing both excitatory and inhibitory neurons. In such networks, activity among excitatory cells results in inhibitory feedback. This feedback, if powerful enough, can stabilize attractors even without a saturating nonlinearity, essentially by stabilizing the equilibrium (above considered unstable) on the steep part of the gain function. The price one pays, though, is that a reasonable fraction of the neurons must be involved in each of the memories, which takes us away from the sparse coding limit and thus reduces network capacity [15].
BibTeX
@inproceedings{latham2001associative,
  author = {Peter E. Latham},
  title = {{Associative memory in realistic neuronal networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2001},
  url = {https://openreview.net/forum?id=r1-FlP-uWH},
}
Figure