Phase Transitions in Neural Networks

Joshua ChoverNeurIPS 1987

Introduction

This paper explores how neural networks undergo "phase transitions"—sudden, dramatic shifts in activity levels—similar to water boiling or freezing. By modeling both infinite and finite cellular arrays, the author demonstrates that adjusting parameters like spontaneous firing rates can push a network from quick extinction into a state of sustained, "metastable" activity. This theoretical framework potentially explains how neuromodulators optimize brain function for memory and response.

The Challenge of Modeling Neural Tipping Points

In the late 1980s, researchers noticed that simulations of brain regions like the piriform cortex and hippocampus didn't just change linearly; they often hit "tipping points." Small changes in a single parameter (like the ratio of inhibitory to excitatory cells) could cause the entire network to flip from near-silence to intense, sustained firing.

Joshua Chover seeks to provide a rigorous mathematical framework for these observations. The core challenge is bridging the gap between infinite models (where phase transitions are mathematically "sharp") and finite models (actual biological or simulated networks), where activity eventually dies out but can persist for a very long time in a "metastable" state.

Modeling Approaches

The author employs two primary modeling approaches to understand these dynamics:

1. The "Simple" Model

This model consists of an infinite linear array of cells governed by discrete-time rules. Firing is determined by a neighbor's activity and a "spontaneous firing" probability ($p$). Chover maps this behavior onto a random walk among non-negative integers. If the probability of the active "stretch" shrinking is higher than it growing, the activity dies out.

1D cellular array model showing activity spread over time

2. The "Complex" Model

A more biologically realistic simulation that introduces several constraints:

  • Refractory periods: Cells cannot fire immediately after a previous spike.
  • Feedback and Shunt Inhibition: Local "brakes" on activity.
  • Propagation Speeds: Delays in signal travel ($v_I$ and $v_E$).
  • Integration Windows: Cells sum inputs over a specific time constant ($s$) to reach a threshold ($\theta$).

For finite arrays, the author uses Markov transition matrices to describe how the system moves between states, eventually settling into "traps" like total extinction or rhythmic flip-flop patterns. The paper relies on computer simulations of a 200-cell array and references prior biological simulations to ground the abstract models.

Key Insights: Random Walks and Metastability

  • The Random Walk of Activity: In the simple model, the length of the firing pattern ($L_n$) behaves like a random walk. The critical threshold is found at $p_c = 1/2$. Below this, the "walk" is biased toward zero (extinction); above it, activity can drift toward infinity.
  • Metastability in Finite Systems: In a finite network, activity must eventually die out or hit a repeating cycle. However, near the critical value, the "expected time until die-out" explodes. The system enters a "metastable equilibrium" where it behaves as if it were in a permanent high-activity phase for a biologically significant duration.
Phase transition graphs showing die-out probability and expected time to extinction
  • Neuromodulation as a Control Knob: The paper conjectures that neuromodulators (like acetylcholine or dopamine) act as the "$p$" parameter. By temporarily raising synaptic efficacy or spontaneous firing levels, the brain can "tune" itself to a near-critical state to amplify responses to stimuli.

Key Results

  • Sharp Transitions: The infinite model shows a classic phase transition where the probability of ultimate die-out drops from 1 to a lower value exactly at the critical point $p_c$.
  • Simulation Divergence: In the complex 200-cell simulation:
  • At $p = 0.10$, activity is sparse and dies out quickly (e.g., by time $n=115$).
  • At $p = 0.20$, the network enters a "relatively rigid patterning" that persists far longer (the simulation was truncated at $n=525$ while still active).
  • Functional Benefits: The author notes that being near-critical allows for "heightened firing response," which leads to faster changes in synaptic efficacy (faster memory storage) and faster access to existing memories.

Limitations and Future Work

  • 1D Geometry: The models are primarily linear (1D) arrays. The author acknowledges that the dependence of responses on initial conditions and parameters is "intricate" and would likely be even more complex in 2D or 3D architectures.
  • The "Bad Effect": Chover warns that while criticality is good for memory, being too close to the edge can lead to "wild, epileptiform activity"—essentially a biological system losing control of its phase transition.
  • Steady-State vs. Transient: The paper focuses on "mid-term transient activity" because, in these Markovian models, the long-run stable firing rate is technically zero. Future work would need to reconcile this with networks that maintain non-zero steady-state activity.

Reference: Chover, J. (1988). Phase Transitions in Neural Networks. University of Wisconsin, Madison. Published by American Institute of Physics.

Summary updated

BibTeX
@inproceedings{chover1987phase,
  author = {Joshua Chover},
  title = {{Phase Transitions in Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1987},
}
Figure