Scaling Laws for Associative Memory Capacity

This direction is merged into another direction: linear memory capacity scaling. Its last version stays readable and citable here.

Version 1 of 117 papers20 conceptsfrozen 2026-10-04

How much information can a neural network store? In the early days of connectionism, the answer was discouraging: the capacity of a standard Hopfield network scales only sub-linearly or, at best, linearly with a very small coefficient relative to the number of neurons. This "capacity bottleneck" suggested that to store more memories, one needed an exponentially larger number of physical components.

However, a diverse body of research—spanning from 1980s VLSI design to modern spiking networks and "fast weight" architectures—reveals that this bottleneck is not a fundamental law of physics, but a consequence of specific architectural choices. By rethinking how synapses store information and how neurons interact, researchers have uncovered scaling laws that allow memory capacity to grow linearly, or even quadratically, with the network's resources.

From Energy Landscapes to Signal Spaces

Traditional associative memories function by minimizing an "energy" function, where memories are local minima. This approach is robust but inefficient. Robert J. Marks II and colleagues introduced Alternating Projection Neural Networks (APNNs) (The Performance of Convex Set Projection Based Neural Networks), which shift the problem from energy minimization to geometric projection in signal space.

In this framework, capacity scales linearly with the total number of neurons. This predictability is a significant leap over energy-based models, where capacity often collapses as the network grows. Similarly, Allan Hartstein and Roger H. Koch demonstrated that hardware-friendly networks leveraging MOSFET physics could achieve linear scaling, proving that efficient associative memory is physically realizable in silicon (A Self-Learning Neural Network).

Overcoming the Square-Root Bottleneck

In many biological models, memory capacity was thought to scale only with the square root of the number of synapses ($\sqrt{N}$). This would mean that a brain twice as large would be significantly less than twice as smart. Marcus K. Benna and Stefano Fusi challenged this in Complex synapses as efficient memory systems.

Their intuition relies on synaptic complexity. Instead of a single weight value, they propose synapses with multiple internal "hidden" variables that interact across different timescales. This allows the synapse to consolidate memories: new information is stored in fast-changing variables, while older, more important information filters down into stable, slow-changing variables. This mechanism allows capacity to scale almost linearly with the number of synapses, a massive improvement that has recently been emulated in hardware using proton-conducting polymers (Li Ji-An et al., Emulating Complex Synapses Using Interlinked Proton Conductors).

The Power of Sparseness and Balance

Why does linear scaling often fail in practice? The answer usually involves noise and interference. When too many neurons fire at once, memories overlap and blur.

  1. Inhibition and Sparseness: Axel Kammerer and Christian Leibold showed that global inhibitory feedback forces "sparse coding," where only a few neurons represent any given memory (Inhibition enhances capacity of sequence replay). This sparseness is the engine of high storage density.
  2. Double Balancing: Yuval Aviel and David Horn introduced the "double balance principle," pairing excitatory memory patterns with inhibitory "shadows" (The Doubly Balanced Network of Spiking Neurons). This maintains the network in a sensitive, asynchronous state, allowing for a capacity of $P \approx 0.06N$.
  3. Low-Rank Manifolds: Modern approaches by Christian K. Machens and William F. Podlaski treat memory as a geometric problem on a hypercubic manifold (Storing overlapping associative memories on latent manifolds in low-rank spiking networks). By restricting connectivity to a low-rank structure, they achieve stable recall for a number of patterns equal to half the neuron count.

Beyond Neurons: Fast Weight Programmers

While most models focus on neuron-based storage, Kazuki Irie and Jürgen Schmidhuber argue that the highest density comes from storing information in the weights themselves (Learning to Control Rapidly Changing Synaptic Connections).

In Fast Weight Programmers (FWPs), one part of the network "writes" to the synaptic weights of another part. Because the number of weights grows quadratically with the number of neurons ($N^2$), this architecture offers a quadratic increase in memory capacity. This insight provides the mathematical foundation for modern Linear Transformers, linking 1990s memory research to today's large-scale AI.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_laws_for_associative_memory_capa_ed8955bd,
  title = {Scaling Laws for Associative Memory Capacity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-laws-for-associative-memory-capacity/rdr_ed8955bd140e403b8c1790248870555e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_laws_for_associative_memory_capa_ed8955bd_v1,
  title = {Scaling Laws for Associative Memory Capacity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-laws-for-associative-memory-capacity/rdr_ed8955bd140e403b8c1790248870555e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}