The Inherent Simplicity Bias of Neural Architectures

This direction is split into other directions: architectural simplicity bias. Its last version stays readable and citable here.

Version 1 of 151 papers83 conceptsfrozen 2026-10-04

In classical learning theory, a model with more parameters than data points should be prone to overfitting—memorizing noise rather than learning patterns. Yet, deep neural networks (DNNs) routinely defy this, generalizing to new data even when they have the capacity to fit arbitrary labels. While many researchers credit the "magic" of Stochastic Gradient Descent (SGD) for this success, a growing body of evidence suggests that generalization is actually "pre-wired" into the architecture itself.

This phenomenon is known as Simplicity Bias: the inherent mathematical tendency of neural networks to represent simple, structured functions over complex, noisy ones.

The Parameter-Function Map is Rigged

To understand why DNNs generalize, we must distinguish between parameter space (the billions of weights in a model) and function space (the actual input-output mapping the model performs).

As Guillermo Valle-Pérez, Ard A. Louis, and Chico Q. Camargo argue in Deep learning generalizes because the parameter-function map is biased towards simple functions, the mapping between these two spaces is highly non-uniform. If you were to pick a set of weights at random, you might expect to get a "random" function that looks like white noise. Instead, you are exponentially more likely to get a simple function.

This is the Volume Hypothesis: simple functions occupy a much larger volume of the parameter space than complex ones. Because many different weight combinations can represent the same simple pattern (a form of structural redundancy), a model is statistically "pushed" toward simplicity before training even begins.

Algorithmic Occam’s Razor

How do we define "simple"? Researchers often turn to Kolmogorov complexity—the length of the shortest program that can produce a specific output.

In The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning, Micah Goldblum and Andrew Gordon Wilson explain that real-world data is not random; it has low Kolmogorov complexity. Crucially, neural networks share this bias. Chris Mingard and colleagues have shown that DNNs are a priori biased towards Boolean functions with low entropy, meaning they naturally prefer highly imbalanced or compressible outputs.

This creates a fortunate alignment: the "Occam's Razor" built into the network's geometry matches the structured nature of the universe.

Neural Redshift and Architectural Folding

The physical construction of a network—its layers, activations, and connectivity—acts as a filter. Guido Montúfar and Yoshua Bengio demonstrated in On the Number of Linear Regions of Deep Neural Networks that deep networks "fold" the input space, reusing computation to create complex boundaries. However, Feng-Lei Fan and Huan Xiong found that in practice, Deep ReLU Networks Have Surprisingly Simple Polytopes. The regions they create are geometrically simple, preventing the model from becoming too "jagged."

This structural preference for low-frequency patterns is what Damien Teney calls Neural Redshift in Random Networks are not Random Functions. Even with completely random weights, networks are "pre-wired" to produce smooth, low-complexity functions. As depth increases, this bias often strengthens, with David Saad noting that random ReLU networks naturally converge toward simpler, constant functions in their deepest layers.

Why SGD Works

If the architecture is already biased toward simplicity, what is the role of the optimizer? Chris Mingard and Guillermo Valle-Pérez suggest in Is SGD a Bayesian sampler? Well, almost that SGD acts less like a sophisticated engineer and more like a Bayesian sampler.

Because simple functions occupy such a massive portion of the parameter space, any optimization process that searches for a solution is statistically overwhelmed by simple ones. SGD doesn't have to "find" a needle in a haystack; it just has to pick a piece of hay, and the hay is almost always a simple, generalizable function.

A Universal Principle of Findability

This simplicity bias isn't unique to AI. In Phenotype bias determines how natural RNA structures occupy the morphospace of all possible shapes, Kamaludin Dingle and Ard A. Louis show that biological systems like RNA also exhibit a massive skew toward a few "simple" shapes.

Whether in biology or deep learning, the conclusion is the same: the way a system is constructed dictates what it can easily "find." For neural networks, this "findability constraint" ensures that even when we give them the power to be complex, they almost always choose to be simple.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_inherent_simplicity_bias_of_neural_a_3e3b264e,
  title = {The Inherent Simplicity Bias of Neural Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-inherent-simplicity-bias-of-neural-architectures/rdr_3e3b264e0f2e4244a271138c7f36e9cc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_inherent_simplicity_bias_of_neural_a_3e3b264e_v1,
  title = {The Inherent Simplicity Bias of Neural Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-inherent-simplicity-bias-of-neural-architectures/rdr_3e3b264e0f2e4244a271138c7f36e9cc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}