Why Parameter Maps Favor Simple Functions

36 concepts24 papers3 proposals

Classical learning theory presents a fundamental paradox: an overparameterized model with more parameters than data points should easily overfit by memorizing noise. Yet, modern deep neural networks generalize remarkably well. While much attention has been paid to the implicit regularization imposed by optimization algorithms like gradient descent, a deeper structural truth has emerged. Even before training begins, the physical construction of neural networks is heavily rigged.

The mapping from a network's parameter space to its function space is profoundly non-uniform. When you randomly sample the weights of a neural network, you do not get a random, chaotic function. Instead, you are exponentially more likely to get an orderly, low-complexity function. This intrinsic phenomenon acts as an inbuilt Occam's razor, aligning the model's structural priors with the orderly nature of real-world data.

The Counting Argument: Parameter Volume vs. Function Complexity

To understand why this bias exists, imagine a vast control panel containing millions of dials (the parameters). Each unique configuration of these dials produces a specific behavior (a function). If you spin all the dials completely at random, what is the probability of hitting a specific behavior?

The answer depends entirely on structural redundancy. Consider a simple function—for instance, one that outputs a constant zero regardless of the input. In a deep network, there are an astronomical number of ways to achieve this behavior: you could zero out the weights of the first layer, the middle layer, or the final layer, or let intermediate layers cancel each other out. Because of this architectural redundancy, a massive volume of the parameter space maps directly to this single, simple function.

Conversely, consider a highly complex, chaotic function that requires a unique, high-frequency response for every single input point. To implement this function, every single dial on your control panel must be set to a precise, highly specific value. A single deviation destroys the behavior. Consequently, only a tiny, microscopic fraction of the parameter space maps to this complex function.

As Buzaglo et al. demonstrate, a uniform distribution over a network's parameter space does not yield a uniform distribution over its functions. Because many different parameter configurations collapse into the same simple behaviors, random sampling naturally favors lower-complexity solutions. Berchenko formalizes this by showing that constructing complex models from random components inherently tilts the resulting function space toward simplicity, contradicting the standard mathematical assumption that most accessible functions are complex.

The Mathematical Foundation: Algorithmic Occam's Razor

This phenomenon can be rigorously quantified using Algorithmic Information Theory (AIT) and Kolmogorov complexity, which measures the length of the shortest computer program that can produce a given output.

Using these tools, Valle Pérez et al. and Mingard et al. proved that the probability of a neural network architecture expressing a specific function decreases exponentially as the function's descriptional complexity increases. This exponential decay perfectly counteracts the combinatorial explosion of possible complex functions that could theoretically fit a dataset.

   Probability of Function
     ▲
     │  █
     │  █
     │  █ █
     │  █ █ █
     │  █ █ █ █ ▄ ▄ ▄ ▄ ▄ ▄ ▄ ▄
     └───────────────────────────►
        Low Complexity   High Complexity
       (Large Volume)    (Tiny Volume)

To verify this independently of training, Goldblum et al. measured the probability assigned to sequences of varying algorithmic complexity by models with purely random weights. They confirmed that randomly initialized models demonstrate an innate, exponential preference for low-complexity patterns. This explains why optimization algorithms find generalizable solutions so easily: the simple solutions occupy such a massive volume of the search space that the optimizer is statistically bound to stumble into them by chance.

How Simplicity Manifests Across Architectures

Depending on the specific architecture of a network, this simplicity bias manifests in distinct mathematical flavors:

  • Low-Rank Representations: By analyzing the geometry of deep networks, Huh et al. showed that depth itself acts as a regularizer. A larger volume of the parameter space in deep architectures corresponds to low effective rank feature representations, meaning the network naturally compresses data into low-dimensional spaces.
  • Low-Degree Polynomials: When networks are trained, this structural bias guides the trajectory of learning. Abbe et al. and Zhang et al. demonstrated that neural networks exhibit an implicit bias toward the lowest-degree polynomial representation that satisfies training constraints. This explains why networks prefer to represent features linearly when possible; linearity is simply the lowest-degree, most parameter-efficient functional form available.
  • Low Input Sensitivity: Certain architectures are structurally insulated against noise. De Palma et al. proved that wide ReLU networks are highly robust to input perturbations, making them structurally incapable of representing highly sensitive functions like parity. Similarly, Bhattamishra et al. showed that Transformers possess an inherent bias toward functions with low sensitivity, predisposing them toward smooth generalizations over high-frequency patterns.

While this bias is largely architectural, it can be augmented. For instance, Zhang et al. show that dropout acts as an explicit simplicity filter by penalizing high-frequency oscillations during training. Conversely, Panwar et al. caution that in context-rich environments like In-Context Learning, a model's preference for simplicity can also be a learned reflection of the pretraining data distribution rather than a purely structural constraint.

A Universal Law of Generative Systems

The bias of parameter-to-function maps is not an isolated quirk of computer science; it appears to be a universal mathematical property governing any complex system where discrete instructions generate physical structures.

In evolutionary biology, the mapping from a genotype (genetic code) to a phenotype (the resulting physical organism) mirrors the parameter-to-function map of a neural network. Johnston et al. and Ahnert et al. discovered that highly symmetric and modular protein complexes spontaneously emerge in nature because they require shorter genetic descriptions. This algorithmic simplicity makes them exponentially more likely to appear via random mutations.

As Martin et al. note, this creates a "developmental bias" where evolution is driven not just by the survival of the fittest, but by the arrival of the frequent. Simple, symmetric forms appear so often as defaults that they can outcompete rarer, more complex structures, even if the complex structures are theoretically more optimal.

  DEEP LEARNING                           BIOLOGY

  Parameter Space (Weights)               Genotype Space (DNA)
         │                                       │
         ▼ [Parameter-Function Map]              ▼ [Genotype-Phenotype Map]
  Function Space (Smooth Curves)          Phenotype Space (Symmetric Shapes)

This universal findability constraint explains why nature uses only a tiny fraction of mathematically possible shapes. Dingle et al. and von der Dunk et al. demonstrated that natural RNA secondary structures are statistically indistinguishable from shapes produced by folding completely random sequences. The prevalence of these shapes is not a masterpiece of adaptive fine-tuning by natural selection, but rather a byproduct of a profound simplicity bias inherent to the physics of folding. Dingle et al. have since shown that this mathematical bound holds true across higher-order biological systems, including tooth development and gene regulation networks.

Ultimately, whether looking at the trajectory of a robot arm optimized by a neural network (Wang et al.) or the shape of an RNA molecule, the underlying principle remains the same. When a system maps a redundant set of parameters to a space of behaviors or shapes, it creates an asymmetric landscape. In that landscape, simplicity is not just an aesthetic preference—it is a statistical guarantee.

Key Foundational Work

Self-assembly, modularity, and physical complexity (2010) — contributed 1 concept elements to this direction.
Free Lunch for Optimisation under the Universal Distribution (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_architectural_simplicity_bias_529dfa4c,
  title = {architectural simplicity bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-simplicity-bias/rdr_529dfa4cc7b847f28d4cbe40e46236c8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_architectural_simplicity_bias_529dfa4c_v1,
  title = {architectural simplicity bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architectural-simplicity-bias/rdr_529dfa4cc7b847f28d4cbe40e46236c8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake