History-Dependent Attractor Neural Networks

Isaac Meilijson, Eytan RuppinNeurIPS 1992

Introduction

This paper introduces a Bayesian framework for Attractor Neural Networks (ANNs) that incorporates "history-dependent" dynamics, moving away from the memoryless nature of traditional Hopfield models. By allowing neurons to integrate past and present inputs and selectively "fire" based on their confidence (a process termed censoring), the authors achieve performance that exceeds the theoretical limits of standard history-independent networks. This approach is particularly effective in biologically realistic settings characterized by sparse connectivity and low neural activity.

The Problem with Memoryless Networks

Traditional Hopfield-style Attractor Neural Networks are often "memoryless"—a neuron's state update depends entirely on the input it receives in the current moment. Furthermore, these models frequently assume dense connectivity and high firing rates. These assumptions contradict the biological reality of the mammalian cortex, where neurons are sparsely connected and overall firing activity remains low.

To bridge this gap, the authors investigate two core questions:

  1. Can network performance be improved by making a neuron's update dependent on its history of inputs?
  2. How should a network decide which neurons "fire" (become active) during an iteration to maximize the accuracy of memory retrieval?

A Bayesian, History-Dependent Approach

The authors employ a Bayesian approach to redefine how neurons update. Instead of relying on a simple threshold function, each neuron calculates the a-posteriori probability that its "true" state should be +1 or -1, given its initial state and the fields (inputs) it has received over time.

Key methodological components include:

  • Generalized Fields: A neuron's decision is based on a linear combination of current and past input fields. This mimics biological "slow-acting" ion channels that allow synaptic inputs to influence a neuron's potential over seconds, rather than just microseconds.
  • Activation Rules (Censoring): The authors test different rules to determine which neurons should participate in an update:
  • Random: Neurons are chosen at random.
  • Tail-censoring: Only neurons with the strongest "beliefs" (highest confidence in their state) fire.
  • Interval-censoring: Neurons with mid-range confidence fire.
  • Hybrid: A combination of tail and interval censoring.
  • Mathematical Modeling: Using a bivariate normal distribution assumption, the authors analytically predict the performance of the network over two iterations, allowing them to calculate an "effective" number of sampled neurons.

Experimental Setup

As is standard for theoretical ANN research of this era, the paper relies on simulated environments rather than external image or text datasets:

  • Synthetic Random Memories: The network is tasked with retrieving $m$ "random memories" consisting of independent and identically distributed (i.i.d.) entries of $\pm 1$.
  • Simulated Cortical Architectures: The authors simulate networks with $N$ up to $10^5$ neurons, varying connectivity ($K$) and activity levels to model "columnar" brain structures.

Key Findings

The Censoring Advantage

Including a history-based term in the neuron's dynamics allows the network to outperform the theoretical upper bound of any history-independent network. In the mammalian cortex, not every neuron fires at once. The paper demonstrates that judiciously choosing which neurons fire based on their post-synaptic potential significantly enhances retrieval.

Final Similarity vs. Connectivity (K) for various activation rules.

Interestingly, while "tail-censoring" (firing when most certain) is biologically intuitive, "interval-censoring" (firing when moderately certain) actually provides more stable performance across a wider range of connectivity levels. Furthermore, the math suggests that it is always better to replace one large update iteration with two smaller, history-dependent ones, implying that the brain's asynchronous nature is a feature, not a bug.

Theoretical Match and Superiority over Hopfield

The authors' mathematical predictions for final similarity (overlap) closely matched their experimental simulations.

Predicted vs. Experimental Performance

As shown in the results, the predicted performance for hybrid censoring (0.975) aligns tightly with the experimental outcome (0.972). More importantly, in sparse, low-activity networks (e.g., $N=1500, K=50$), the history-dependent Bayesian approach achieved significantly higher similarity than standard Hopfield dynamics (0.972 vs. 0.902).

Ultimately, the model finds that optimal performance occurs when the number of firing neurons closely corresponds with biological evidence—roughly 4% of neurons in a module firing at any given moment.

Limitations and Future Work

  • Two-Iteration Limit: The analytical framework specifically describes the first two iterations of the network. While these are the most critical for initial retrieval, the long-term steady-state behavior is not fully explored.
  • Gaussian Assumption: The model relies on a "working assumption" that the input fields follow a bivariate normal distribution. While this holds for large $N$, it may break down in extremely small or highly structured networks.
  • Signal Function Complexity: While the paper identifies that hybrid signaling (where some neurons signal the opposite of their belief) is mathematically optimal, the authors acknowledge that the biological plausibility of such a mechanism requires further investigation.

Summary updated

BibTeX
@inproceedings{meilijson1992history,
  author = {Isaac Meilijson and Eytan Ruppin},
  title = {{History-Dependent Attractor Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1992},
}
Figure