Much of our theoretical understanding of neural networks rests on a convenient assumption: that the signals passing through a network—the "pre-activations"—follow a Gaussian (normal) distribution. This assumption is the bedrock of Gaussian Process limits, modern initialization schemes, and uncertainty quantification.
However, as research into finite-width and deep architectures progresses, it has become clear that this "Gaussian Ideal" is fragile. As signals propagate through successive layers, they undergo a process of distributional drift, moving away from the bell curve and toward complex, heavy-tailed distributions that challenge training stability and model reliability.
The Mechanics of Distributional Drift
In an infinitely wide network, the Central Limit Theorem suggests that pre-activations should be Gaussian. But real-world networks are finite. Boris Hanin shows in Correlation Functions in Random Fully Connected Neural Networks at Finite Width that the "effective depth" of a network is governed by the ratio of its depth to its width ($L/n$).
As this ratio increases, deviations from Gaussian behavior accumulate. Even if a network starts with perfectly Gaussian weights, the finite width introduces small correlations that grow polynomially with depth. Eventually, these "finite-width corrections" dominate the statistical output, causing gradients to explode or vanish and breaking the assumptions used to initialize the model.
Training further compounds this issue. While random, untrained networks might approximate Gaussian behavior, Joachim Sicking and colleagues demonstrate in Characteristics of Monte Carlo Dropout in Wide Neural Networks that the learning process introduces strong dependencies. These dependencies warp the activations into non-Gaussian distributions with heavy exponential tails, particularly in the deeper layers of the architecture.
Why Non-Gaussianity Matters
When the Gaussian assumption fails, the tools we use to manage deep networks begin to break. This manifests in three primary ways:
- Uncertainty Miscalculation: Christos Tzelepis and Ioannis Patras note in Uncertainty Propagation in Convolutional Neural Networks: Technical Report that non-linear layers like ReLU and Sigmoid warp probability distributions. If we assume uncertainty remains Gaussian throughout the network, our estimates of "how much the model knows" become increasingly unrepresentative of reality.
- Normalization Failure: Standard techniques like LayerNorm or BatchNorm focus on fixing the mean and variance but ignore the shape of the distribution. In Unified Normalization for Accelerating and Stabilizing Transformers, Qiming Yang and co-authors point out that deep Transformers often develop extreme outliers and volatile statistics. These "non-normal" fluctuations make it difficult for offline normalization to stabilize training, as moving averages become corrupted by rare but extreme values.
- Generalization and Stability: Daniel Eftekhari and Vardan Papyan argue in On the Importance of Gaussianizing Representations that this drift away from normality negatively impacts how well a model generalizes to new data.
Restoring the Bell Curve
If the loss of Gaussianity is the problem, can we force the network to stay "normal"? Researchers are exploring two main avenues: architectural constraints and explicit "Gaussianizing" layers.
Engineering Gaussianity
Pierre Wolinski and Julyan Arbel propose a structural solution in Imposing Gaussian Pre-Activations in a Neural Network. They prove that by pairing a specific weight distribution (Symmetric Weibull) with custom-engineered activation functions, one can mathematically guarantee that pre-activations remain Gaussian at any depth, even in narrow networks. This prevents the development of heavy tails and stabilizes the signal.
Normality Normalization
A more flexible approach involves explicitly transforming activations back into a Gaussian shape during the forward pass. In On the Importance of Gaussianizing Representations, the authors introduce Normality Normalization. Unlike standard normalization, which only shifts and scales data, this layer uses power transforms to reshape the distribution itself. By "Gaussianizing" the representations and adding controlled noise, they find significant improvements in robustness and generalization.
The Resilience of Gaussian Theory
Despite these challenges, the Gaussian framework remains surprisingly robust. Hawoong Jeong and Jaeyong Bae describe a "Gaussian universality" in Gaussian Universality in Neural Network Dynamics with Generalized Structured Input Distributions. They show that even when input data is highly non-Gaussian (like complex images), the learning dynamics often converge to the same behavior as if the data were simple Gaussian noise.
The frontier of this research lies in understanding the tension between this universality and the depth-dependent drift. While the dynamics may look Gaussian, the representations clearly do not. Bridging this gap—either by managing the drift or by forcing the network back toward the bell curve—is becoming essential for building deeper, more reliable models.