Optimal Signalling in Attractor Neural Networks

Isaac Meilijson, Eytan RuppinNeurIPS 1993

Overview

This paper demonstrates that the mathematically optimal way for a neuron to signal its state in a Hopfield-like network is through a "slanted sigmoidal" function that accounts for its own firing history. By using a Bayesian framework to derive this non-monotone activation function, the authors show that networks can achieve higher memory capacity and better retrieval performance, mimicking the biological phenomenon of neural adaptation.

The Problem: Memoryless vs. History-Dependent Neurons

In classical Attractor Neural Networks (ANNs), like the Hopfield model, neurons are typically "memoryless." Their output is a simple function of their current input field, usually a sign function or a standard sigmoid. However, real cortical neurons are much more complex; their response to a stimulus depends heavily on their recent firing history and the presence of neuromodulators.

The authors address the gap between these simplified models and biological reality. They ask: What is the computationally most efficient way for a neuron to signal its state if it can "remember" its own history? Specifically, they look at how history-dependent dynamics can improve the network's ability to converge on a stored memory pattern (the "attractor") within just a few iterations.

Methods: A Bayesian Framework for Neural Dynamics

The researchers employ a Bayesian framework to model neuronal dynamics. Instead of a simple threshold, each neuron treats its state as a statistical inference problem. It calculates the a-posteriori probability of belonging to a specific memory pattern based on:

  1. The initial input signal it received.
  2. The "field" (sum of signals) it receives from other neurons.
  3. Its own firing history.

The core innovation is the derivation of the optimal analog signal function. They prove that the best function is a "slanted sigmoid," defined as: $$R(s, t) = A_4 \tanh(s) - A_5(s - g(t))$$ Where $s$ is the current input and $g(t)$ represents the prior information. This function is non-monotone, meaning that as the input field increases beyond a certain point, the output signal actually decreases.

The optimal analog signal function (a) showing a non-monotone "slanted sigmoid" for active and silent neurons, and its discretized multi-threshold counterpart (b).

Key Insights: Adaptation and Network Architecture

  • The "Slanted Sigmoid" and Adaptation: The most striking finding is that the optimal signal function isn't a simple "S-curve." It's non-monotone. This means neurons that have already fired strongly in the first iteration have a "lesser tendency" to send positive signals in the next. This mathematically mirrors neural adaptation in biology, where a neuron's sensitivity decreases after prolonged activity.
  • Sparse is Better: The paper finds that for a fixed number of connections per neuron ($K$), a large, sparsely connected network performs better than a smaller, fully connected one. This provides a computational justification for the sparse connectivity seen in the mammalian cortex.
  • Quality over Quantity in Inputs: The authors show that it is more efficient to provide high-fidelity information to a small subset of "input" neurons than to provide a low-fidelity, distorted version of the memory to the entire network.
  • Capacity Boost: By using these history-dependent, non-monotone dynamics, the network can successfully retrieve memories even when the "memory load" (the ratio of stored patterns to connections) is significantly higher than the theoretical limit of standard Hopfield networks.

Key Results: Superior Retrieval Performance

The optimal history-dependent dynamics significantly outperformed previous "memoryless" or heuristic Bayesian models. In simulations on networks of $N=5000$ neurons, patterns were "signalled" by setting a subset of neurons to a state with a specific level of similarity to a stored memory.

Performance comparison showing Final Similarity vs. Connectivity (K). Analog and discretized optimal signalling rapidly approach near-perfect similarity compared to standard posterior-probability methods.
  • Near-Perfect Retrieval: In simulations where only 4% of neurons received an initial signal, the network still achieved near-perfect final similarity ($>0.98$) to the correct memory pattern within two iterations, provided connectivity was sufficiently high ($K > 1000$).
  • Biological Alignment: The derived "bi-threshold" behavior—where only neurons within a specific range of input field values fire—aligns with observed biological phenomena where neurons with very high field values are silenced due to adaptation.

Limitations and Future Work

  • Two-Iteration Focus: The analytical results primarily focus on the performance after two iterations. While the authors use Martingale arguments to suggest similarity increases persistently, a full multi-iteration analysis is not detailed.
  • Simplified Synchrony: The model assumes a degree of synchrony ("iterations") that may not perfectly capture the continuous-time, asynchronous nature of real biological circuits.
  • Future Directions: The authors suggest that further investigation into the 0,1 (silent/firing) formulation could reveal more about the "bizarre" but biologically relevant behavior of multi-threshold neurons.

Reference: Meilijson, I., & Ruppin, E. (1994). Optimal signalling in Attractor Neural Networks. Advances in Neural Information Processing Systems (NIPS).

Summary updated

BibTeX
@inproceedings{meilijson1993optimal,
  author = {Isaac Meilijson and Eytan Ruppin},
  title = {{Optimal Signalling in Attractor Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1993},
}
Figure