Linear Memory Capacity Scaling in Neural Networks

27 concepts21 papers1 proposal

How much information can a neural network store? Early theoretical models of associative memory, such as dense Hopfield networks, suggested frustratingly low limits, where memory capacity scales sub-linearly or logarithmically unless the network is massively over-parameterized. However, a rich line of research reveals that under the right structural conditions—namely sparsity, local synaptic constraints, and excitation-inhibition balance—a network's memory capacity can scale linearly, or even superlinearly, with its size.

To understand why this works, we must shift our perspective from viewing a network as a massive, unconstrained optimization space to viewing it as a structured geometric engine. By managing how signals are encoded and how synapses are bounded, we can drastically simplify the geometric constraints required to store and retrieve distinct patterns.

The Geometry of Linear Scaling

In a standard associative memory system, storing a pattern means creating an attractor or a stable state in the network's energy landscape. If we use dense, unconstrained representations, adding new memories quickly corrupts existing ones due to cross-talk.

However, if we change the underlying signal space, the scaling behavior transforms. For instance, Robert J. Marks II et al. demonstrated in The Performance of Convex Set Projection Based Neural Networks that Alternating Projection Neural Networks (APNNs) can achieve a storage capacity that scales linearly with the total number of input and hidden neurons. By replacing energy minimization with a signal-space approach based on convex set projections, the network avoids sub-linear bottlenecks.

Similarly, even when implementing hardware-friendly constraints—such as replacing multiplicative synapses with MOSFET threshold functions—Allan Hartstein et al. in A Self-Learning Neural Network showed that associative memory capacity maintains a predictable linear scaling relationship with the number of neurons.

Why Sparsity Unlocks Near-Optimal Storage

The most powerful mechanism for achieving linear or near-optimal capacity scaling is sparsity. Intuition might suggest that reducing the number of active neurons per pattern reduces the total information the network can carry. While true in an absolute sense, sparsity drastically simplifies the geometric requirements for linear separability.

Pierre Baldi et al. proved in A theory of capacity and sparse neural encoding that increasing the sparsity of a target layer significantly boosts storage capacity. By reducing active components, the network can reliably store far more distinct associations using simple, local learning rules like Hebbian learning.

In the context of optical communications, Mario P. Vecchi et al. showed in Neuromorphic Networks Based on Sparse Optical Orthogonal Codes that sparse binary encodings allow unipolar networks to reach a storage capacity of roughly half the code length, completely bypassing the logarithmic scaling limits of dense models.

This principle is also visible in biological architectures. Nicolas Brunel, in Optimal Properties of Analog Perceptrons with Excitatory Weights, demonstrated a fundamental linear relationship between the fraction of inactive (silent) synapses and total storage capacity: the sum of these two variables remains constant across various distributions. However, sparsity cannot be increased indefinitely without a trade-off; Ryota Miyata et al. noted in Sparsely Encoded Hopfield Model with Unit Replacement that when networks undergo unit replacement (neurogenesis), there exists a specific, optimal level of encoding sparseness that maximizes storage capacity, rather than a monotonic increase.

Local Connectivity vs. Global Population Size

When scaling up to massive, multimodular networks, a common bottleneck emerges: a network's functional performance is often constrained by the number of connections per individual neuron rather than the total number of neurons in the system. As Daniel Martí et al. explained in Scalability properties of multimodular networks with dynamic gating, if connectivity remains sparse while the network grows, a scalability bottleneck occurs.

To resolve this, we must look at how capacity scales relative to local synaptic density. Yasser Roudi demonstrated in A Balanced Memory Network that the storage capacity of a sparsely connected spiking network scales linearly with the number of excitatory connections per neuron, rather than the total population size. This allows large-scale networks to scale their storage capabilities based on local density while maintaining stable firing rates.

Superlinear Capacity via Dynamical Compressed Sensing

Can a network store more patterns than its physical node count? If the input signals are sparse over time, the answer is yes. By framing recurrent network dynamics as a compressive measurement process, we can leverage the mathematics of compressed sensing.

Surya Ganguli et al. showed in Short-term memory in neuronal networks through dynamical compressed sensing that the duration for which a network can retain information scales inversely with the sparsity of the input signal. When the fraction of active components decreases, the theoretical memory capacity increases dramatically.

Expanding on this, Adam S. Charles et al. proved in Short Term Memory Capacity in Networks via the Restricted Isometry Property that recurrent networks can achieve superlinear memory capacity. This occurs because the effective measurement matrix—derived from the powers of the connectivity matrix—satisfies the Restricted Isometry Property (RIP). Crucially, the authors showed that this capability is largely independent of the specific network topology (whether modular or small-world), provided the connectivity matrix maintains approximate orthogonality.

Balancing and Normalization: Keeping Dynamics Stable

To achieve linear capacity scaling without causing catastrophic interference or chaotic firing rates, networks require explicit balancing mechanisms.

One approach is local inhibitory balancing. Yuval Aviel et al. showed in The Doubly Balanced Network of Spiking Neurons: A Memory Model with High Capacity that pairing every excitatory memory pattern with a "shadow" inhibitory pattern allows memory capacity to scale linearly ($P = 0.06N$) while preserving an asynchronous, brain-like state.

Another approach involves synaptic weight normalization. Gal Chechik et al. demonstrated in Effective Learning Requires Neuronal Remodeling of Hebbian Synapses that when synaptic weights are subject to a normalization constraint, memory capacity shifts from a bounded regime to a linear scaling regime. Furthermore, Bruce Graham et al. showed in Capacity and Information Efficiency of a Brain-like Associative Net that normalizing input signals by total activity levels compensates for variations in physical connectivity, allowing partially connected systems to maximize their information efficiency.

Bridging Theory and Practice in Deep Networks

While classical associative networks provide clean analytical models, do these linear scaling laws hold for modern feedforward networks trained via gradient descent?

Historically, deep learning theory suggested that networks must be highly over-parameterized (e.g., quadratically) to guarantee perfect memorization of a dataset. However, Amit Daniely closed this gap in Neural Networks Learning and Memorization with (almost) no Over-Parameterization, proving that standard Stochastic Gradient Descent (SGD) can achieve perfect training set memorization using a parameter count that scales nearly linearly with the number of data points, matching information-theoretic lower bounds.

Of course, the exact upper bounds are strictly governed by architecture. Pierre Baldi et al. established in The capacity of feedforward neural networks that the exact information storage capacity of feedforward networks scales cubically relative to layer sizes (following the sum of products of adjacent layer dimensions), highlighting how bottleneck layers act as structural regularizers. Finally, these stored memories leave an indelible mark on the network structure itself: Sebastian Goldt et al. showed in Bayesian reconstruction of memories stored in neural networks from their connectivity that stored patterns can be reverse-engineered directly from the noisy, rectified connectivity matrix, provided the effective signal-to-noise ratio satisfies specific statistical physics thresholds.

Key Foundational Work

A Self-Learning Neural Network (1988) — contributed 1 concept elements to this direction.
The Performance of Convex Set Projection Based Neural Networks (1987) — contributed 1 concept elements to this direction.
Neuromorphic Networks Based on Sparse Optical Orthogonal Codes (1987) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_memory_capacity_scaling_9033f87c,
  title = {linear memory capacity scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-memory-capacity-scaling/rdr_9033f87cdd33413a885b59dce37202d9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_memory_capacity_scaling_9033f87c_v1,
  title = {linear memory capacity scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-memory-capacity-scaling/rdr_9033f87cdd33413a885b59dce37202d9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake