A Generative Model for Attractor Dynamics

Richard S. Zemel, Michael MozerNeurIPS 1999 · OpenReview

Introduction

Attractor networks are a staple of cognitive science and artificial intelligence, primarily used for pattern completion—the process of taking a noisy or incomplete input and "pulling" it toward a known, clean memory (the attractor). Imagine a ball rolling down a hill into a valley; in this analogy, the valley is the attractor.

However, traditional attractor networks, such as Hopfield nets, are notoriously difficult to work with. Because they store memories in a distributed manner—where every connection in the network helps store every memory—different memories often interfere with one another. This interference leads to two major headaches:

  1. Spurious Attractors: The network creates "fake" memories (unintended valleys in the energy landscape) that were never present in the training data.
  2. Design Difficulty: It is incredibly hard to "sculpt" the energy landscape to have specific properties, such as making one memory easier to recall than another.

A Localist, Generative Approach

To solve these issues, the authors propose a localist attractor network. Instead of distributing knowledge across all connections, they assign a specific unit to each attractor.

To give this structure mathematical rigor, they frame it as a generative model based on a three-level hierarchy:

  1. Select an Attractor ($x$): Pick a discrete category (e.g., the concept of a "dog").
  2. Select a State ($y$): Generate an idealized version of that category based on the attractor's location in the state space.
  3. Select an Observation ($z$): Add noise to that state to create the actual input the network sees.

The network’s dynamics are then derived as an iterative search for the maximum likelihood state. By using a mean-field approximation to minimize "free energy," the authors derive update equations that resemble a sophisticated Expectation-Maximization (EM) algorithm. As the network iterates, it moves the current state closer to the most likely attractor while simultaneously "sharpening" its focus by reducing internal noise parameters.

Key Insights: The "Gang Effect" and Spurious States

One of the most interesting findings is that localist networks naturally exhibit "gang effects." In this phenomenon, a cluster of similar attractors creates a much deeper and wider basin of attraction than a single isolated one. This mimics human psychological data where similar items reinforce each other.

Energy landscape illustrating the gang effect

Furthermore, this generative model effectively eliminates spurious states. In traditional models, the space between two attractors often creates a third, unintended valley. Here, the update rules tie the state's movement to the "responsibility" (weight) of specific attractors. As the network settles, it effectively "turns off" the influence of distant attractors, preventing the creation of spurious mid-point states.

Experimental Results

The paper evaluates the model using three distinct simulations, demonstrating its robustness and utility:

  • Robustness to Distortion (High-Dimensional Hypercube): The authors tested a synthetic 200-dimensional state space with 200 randomly placed attractors at the corners of a hypercube. The network was remarkably stable. As shown in the results, spurious responses were exceedingly rare until more than 80-85% of the input features were missing or distorted.
Spurious responses vs. missing features
  • Winning the Tug-of-War (Face Images): Using a dataset of face images, including "morphed" faces (linear combinations of two different faces), the authors studied how the network chooses between competing memories. The "gang effect" proved powerful: if an input was a morph between a single face and a "gang" of similar faces, the network would settle on the gang face even if the initial input was weighted 60% toward the single face.
  • Word Retrieval (3-Letter English Words): The model successfully functioned as a content-addressable memory using a 78-dimensional dataset of 423 three-letter words. When queried with "DEG" (a non-word), the network used the gang effect of similar words (like BEG, PEG, DOG) to settle on a valid English word that shared the most "support" from its neighborhood.

Limitations and Future Work

The primary limitation acknowledged by the authors is compositional structure. The localist approach requires one unit per attractor. If you are trying to model a world where things are built from parts (like sentences built from words), the number of required units would explode exponentially.

While this model is excellent for pattern recognition and associative memory where the number of items is manageable, future work will need to address how to handle more complex, hierarchical representations without losing the clean statistical properties of the localist approach.

Summary updated

Abstract

Attractor networks, which map an input space to a discrete out(cid:173) put space, are useful for pattern completion. However, designing a net to have a given set of attractors is notoriously tricky; training procedures are CPU intensive and often produce spurious afuac(cid:173) tors and ill-conditioned attractor basins. These difficulties occur because each connection in the network participates in the encod(cid:173) ing of multiple attractors. We describe an alternative formulation of attractor networks in which the encoding of knowledge is local, not distributed. Although localist attractor networks have similar dynamics to their distributed counterparts, they are much easier to work with and interpret. We propose a statistical formulation of localist attract or net dynamics, which yields a convergence proof and a mathematical interpretation of model parameters. Attractor networks map an input space, usually continuous, to a sparse output space composed of a discrete set of alternatives. Attractor networks have a long history in neural network research. Attractor networks are often used for pattern completion, which involves filling in missing, noisy, or incorrect features in an input pattern. The initial state of the attractor net is typically determined by the input pattern. Over time, the state is drawn to one of a predefined set of states-the attractors. Attractor net dynam(cid:173) ics can be described by a state trajectory (Figure 1a). An attractor net is generally implemented by a set of visible units whose activity represents the instantaneous state, and optionally, a set of hidden units that assist in the computation. Attractor dynamics arise from interactions among the units. In most formulations of afuac(cid:173) tor nets,2,3 the dynamics can be characterized by gradient descent in an energy landscape, allowing one to partition the output space into attractor basins. Instead of homogeneous attractor basins, it is often desirable to sculpt basins that depend on the recent history of the network and the arrangement of attractors in the space. In psychological models of human cognition, for example, priming is fundamental: after the model visits an attractor, it should be faster to fall into the same attractor in the near future, i.e., the attractor basin should be broadened. 1 ,6 Another property of attractor nets is key to explaining behavioral data in psycho(cid:173) logical and neurobiological models: the gang effect, in which the strength of an attractor is influenced by other attractors in its neighborhood. Figure 1b illustrates the gang effect: the proximity of the two rightmost afuactors creates a deeper at(cid:173) tractor basin, so that if the input starts at the origin it will get pulled to the right. A Generative Model for Attractor Dynamics 81
BibTeX
@inproceedings{zemel1999generative,
  author = {Richard S. Zemel and Michael Mozer},
  title = {{A Generative Model for Attractor Dynamics}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1999},
  url = {https://openreview.net/forum?id=SkNKIDWuWB},
}
Figure